This is a HTML-ification of Workflow state (workflow.state) and
the associated files in the workflow output directory.
Below is the doc section of the workflow (.workflow file).
Produce, transform and plot fio-output ====================================== This workflow assumes that the 'prep.workflow' has completed. The comparison and plots consists of * bandwidth in KiB / seconds * latency is mean and in nano-seconds * IOPS are 'raw' / base-unit The above in a bar-plot at iodepth=1, and a line-plot as a function of iodepth. See Artifacts for the generated plots
If a workflow step (worklet / run) or test produced any artifacts, then they are listed here.
This is the configuration (.config file) with any
variable-placeholders such as {{ local.env.HOME }} filled out.
devices:
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme0n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme1n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme2n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme3n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme4n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme5n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme6n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme7n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng0n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng1n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng2n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng3n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng4n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng5n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng6n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng7n1
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:41:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:42:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:43:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:44:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:45:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:46:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:47:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:48:00.0'
fio:
bin: /opt/fio/bin/fio
build:
prefix: /opt/fio
engines:
io_uring:
type: builtin
io_uring_cmd:
type: builtin
libaio:
type: builtin
spdk_bdev:
path: /root/git/spdk/build/fio/spdk_bdev
type: external_preload
spdk_nvme:
path: /root/git/spdk/build/fio/spdk_nvme
type: external_preload
xnvme:
path: /usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so
type: external_dynamic
repository:
path: /root/git/fio
remote: https://github.com/axboe/fio.git
tag: fio-3.32
liburing:
repository:
path: /root/git/liburing
remote: https://github.com/axboe/liburing.git
tag: liburing-2.2
os:
name: debian
version: bullseye
spdk:
build:
prefix: /opt/spdk
repository:
branch: for_bdev_xnvme_v6
path: /root/git/spdk
remote: https://github.com/safl/spdk.git
transport:
ssh:
hostname: fabio
port: 22
username: root
xnvme:
build:
type: debug
repository:
path: /root/git/xnvme
remote: https://github.com/OpenMPDK/xNVMe.git
tag: v0.6.0
Steps uses worklets. A description of the worklet used by
the current step is provided below. The description is extracted from the docstring
of the worklet and provided here in verbatim / as-is.
Run fio...
{'description': '\n Run fio...\n',
'extras': {'runlog': {'cmd_01': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_01.output'),
'state': {'begin': 1666781262.0675218,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8211526870727539,
'end': 1666781262.8886745,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_01.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_01.state')},
'cmd_02': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'150921.39 IOPS 73.69 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'150921.39 IOPS 73.69 MiB/s\n'
'[2022-10-26 14:45:55.003669] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:45:55.003716] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:45:55.003722] bdevperf '
'[2022-10-26 14:45:55.003727] '
'--no-shconf [2022-10-26 '
'14:45:55.003731] -l 0 [2022-10-26 '
'14:45:55.003736] --huge-unlink '
'[2022-10-26 14:45:55.003740] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:45:55.003744] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:45:55.003748] '
'--log-level=user1:6 [2022-10-26 '
'14:45:55.003753] --iova-mode=pa '
'[2022-10-26 14:45:55.003759] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:45:55.003763] '
'--match-allocations [2022-10-26 '
'14:45:55.003768] '
'--file-prefix=spdk_pid3400 '
'[2022-10-26 14:45:55.003772] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:45:55.125466] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:45:55.168002] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:45:55.168555] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_02.output'),
'state': {'begin': 1666781264.0326226,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.173058986663818,
'end': 1666781275.2056816,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_02.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_02.state')},
'cmd_03': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_03.output'),
'state': {'begin': 1666781275.208141,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7913839817047119,
'end': 1666781275.999525,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_03.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_03.state')},
'cmd_04': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'171436.90 IOPS 83.71 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'171436.90 IOPS 83.71 MiB/s\n'
'[2022-10-26 14:46:08.004494] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:46:08.004535] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:46:08.004541] bdevperf '
'[2022-10-26 14:46:08.004546] '
'--no-shconf [2022-10-26 '
'14:46:08.004550] -l 0 [2022-10-26 '
'14:46:08.004555] --huge-unlink '
'[2022-10-26 14:46:08.004559] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:46:08.004564] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:46:08.004568] '
'--log-level=user1:6 [2022-10-26 '
'14:46:08.004573] --iova-mode=pa '
'[2022-10-26 14:46:08.004577] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:46:08.004588] '
'--match-allocations [2022-10-26 '
'14:46:08.004593] '
'--file-prefix=spdk_pid3427 '
'[2022-10-26 14:46:08.004597] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:46:08.126140] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:46:08.166933] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:46:08.167491] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_04.output'),
'state': {'begin': 1666781277.113264,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.077265501022339,
'end': 1666781288.1905296,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_04.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_04.state')},
'cmd_05': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_05.output'),
'state': {'begin': 1666781288.1932762,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8355164527893066,
'end': 1666781289.0287926,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_05.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_05.state')},
'cmd_06': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'159095.18 IOPS 77.68 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'159095.18 IOPS 77.68 MiB/s\n'
'[2022-10-26 14:46:21.013312] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:46:21.013360] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:46:21.013365] bdevperf '
'[2022-10-26 14:46:21.013370] '
'--no-shconf [2022-10-26 '
'14:46:21.013374] -l 0 [2022-10-26 '
'14:46:21.013379] --huge-unlink '
'[2022-10-26 14:46:21.013383] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:46:21.013387] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:46:21.013391] '
'--log-level=user1:6 [2022-10-26 '
'14:46:21.013396] --iova-mode=pa '
'[2022-10-26 14:46:21.013400] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:46:21.013404] '
'--match-allocations [2022-10-26 '
'14:46:21.013408] '
'--file-prefix=spdk_pid3453 '
'[2022-10-26 14:46:21.013412] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:46:21.135461] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:46:21.187017] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:46:21.187578] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_06.output'),
'state': {'begin': 1666781290.0950124,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.119955778121948,
'end': 1666781301.2149682,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_06.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_06.state')},
'cmd_07': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_07.output'),
'state': {'begin': 1666781301.2178962,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.9140396118164062,
'end': 1666781302.1319358,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_07.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_07.state')},
'cmd_08': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'137262.72 IOPS 67.02 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'137262.72 IOPS 67.02 MiB/s\n'
'[2022-10-26 14:46:34.208092] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:46:34.208139] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:46:34.208145] bdevperf '
'[2022-10-26 14:46:34.208150] '
'--no-shconf [2022-10-26 '
'14:46:34.208154] -l 0 [2022-10-26 '
'14:46:34.208158] --huge-unlink '
'[2022-10-26 14:46:34.208164] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:46:34.208168] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:46:34.208172] '
'--log-level=user1:6 [2022-10-26 '
'14:46:34.208177] --iova-mode=pa '
'[2022-10-26 14:46:34.208181] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:46:34.208187] '
'--match-allocations [2022-10-26 '
'14:46:34.208191] '
'--file-prefix=spdk_pid3478 '
'[2022-10-26 14:46:34.208195] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:46:34.330276] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:46:34.370957] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:46:34.371536] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_08.output'),
'state': {'begin': 1666781303.3105567,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.075981140136719,
'end': 1666781314.3865378,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_08.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_08.state')},
'cmd_09': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_09.output'),
'state': {'begin': 1666781314.3892505,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8424861431121826,
'end': 1666781315.2317367,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_09.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_09.state')},
'cmd_10': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'138058.37 IOPS 67.41 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'138058.37 IOPS 67.41 MiB/s\n'
'[2022-10-26 14:46:47.231816] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:46:47.231858] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:46:47.231864] bdevperf '
'[2022-10-26 14:46:47.231868] '
'--no-shconf [2022-10-26 '
'14:46:47.231872] -l 0 [2022-10-26 '
'14:46:47.231877] --huge-unlink '
'[2022-10-26 14:46:47.231881] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:46:47.231885] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:46:47.231889] '
'--log-level=user1:6 [2022-10-26 '
'14:46:47.231893] --iova-mode=pa '
'[2022-10-26 14:46:47.231898] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:46:47.231902] '
'--match-allocations [2022-10-26 '
'14:46:47.231906] '
'--file-prefix=spdk_pid3503 '
'[2022-10-26 14:46:47.231910] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:46:47.353497] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:46:47.394950] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:46:47.395495] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_10.output'),
'state': {'begin': 1666781316.3184137,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.08516263961792,
'end': 1666781327.4035764,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_10.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_10.state')},
'cmd_100': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'482862.92 IOPS 235.77 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'482862.92 IOPS 235.77 MiB/s\n'
'[2022-10-26 14:56:34.681870] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:56:34.681913] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:56:34.681919] bdevperf '
'[2022-10-26 14:56:34.681924] '
'--no-shconf [2022-10-26 '
'14:56:34.681928] -l 0 '
'[2022-10-26 14:56:34.681932] '
'--huge-unlink [2022-10-26 '
'14:56:34.681937] '
'--log-level=lib.eal:6 '
'[2022-10-26 14:56:34.681941] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:56:34.681945] '
'--log-level=user1:6 [2022-10-26 '
'14:56:34.681949] --iova-mode=pa '
'[2022-10-26 14:56:34.681952] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:56:34.681957] '
'--match-allocations [2022-10-26 '
'14:56:34.681961] '
'--file-prefix=spdk_pid4648 '
'[2022-10-26 14:56:34.681966] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:56:34.804031] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:56:34.842979] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 14:56:34.843588] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_100.output'),
'state': {'begin': 1666781903.7373946,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.132895708084106,
'end': 1666781914.8702903,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_100.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_100.state')},
'cmd_101': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_101.output'),
'state': {'begin': 1666781914.877601,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8195874691009521,
'end': 1666781915.6971884,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_101.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_101.state')},
'cmd_102': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'482460.34 IOPS 235.58 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'482460.34 IOPS 235.58 MiB/s\n'
'[2022-10-26 14:56:47.633077] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:56:47.633125] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:56:47.633131] bdevperf '
'[2022-10-26 14:56:47.633136] '
'--no-shconf [2022-10-26 '
'14:56:47.633141] -l 0 '
'[2022-10-26 14:56:47.633146] '
'--huge-unlink [2022-10-26 '
'14:56:47.633150] '
'--log-level=lib.eal:6 '
'[2022-10-26 14:56:47.633155] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:56:47.633160] '
'--log-level=user1:6 [2022-10-26 '
'14:56:47.633164] --iova-mode=pa '
'[2022-10-26 14:56:47.633169] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:56:47.633174] '
'--match-allocations [2022-10-26 '
'14:56:47.633178] '
'--file-prefix=spdk_pid4673 '
'[2022-10-26 14:56:47.633183] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:56:47.755367] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:56:47.798960] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 14:56:47.799547] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_102.output'),
'state': {'begin': 1666781916.7448435,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.084971189498901,
'end': 1666781927.8298147,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_102.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_102.state')},
'cmd_103': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_103.output'),
'state': {'begin': 1666781927.8339686,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8341481685638428,
'end': 1666781928.6681168,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_103.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_103.state')},
'cmd_104': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'395076.98 IOPS 192.91 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'395076.98 IOPS 192.91 MiB/s\n'
'[2022-10-26 14:57:00.627211] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:57:00.627256] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:57:00.627262] bdevperf '
'[2022-10-26 14:57:00.627267] '
'--no-shconf [2022-10-26 '
'14:57:00.627271] -l 0 '
'[2022-10-26 14:57:00.627276] '
'--huge-unlink [2022-10-26 '
'14:57:00.627280] '
'--log-level=lib.eal:6 '
'[2022-10-26 14:57:00.627283] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:57:00.627287] '
'--log-level=user1:6 [2022-10-26 '
'14:57:00.627292] --iova-mode=pa '
'[2022-10-26 14:57:00.627297] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:57:00.627301] '
'--match-allocations [2022-10-26 '
'14:57:00.627305] '
'--file-prefix=spdk_pid4698 '
'[2022-10-26 14:57:00.627309] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:57:00.749257] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:57:00.791057] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 14:57:00.791650] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_104.output'),
'state': {'begin': 1666781929.7034733,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.11667513847351,
'end': 1666781940.8201485,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_104.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_104.state')},
'cmd_105': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_105.output'),
'state': {'begin': 1666781940.8240077,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8417377471923828,
'end': 1666781941.6657455,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_105.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_105.state')},
'cmd_106': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'392195.29 IOPS 191.50 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'392195.29 IOPS 191.50 MiB/s\n'
'[2022-10-26 14:57:13.666086] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:57:13.666140] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:57:13.666148] bdevperf '
'[2022-10-26 14:57:13.666155] '
'--no-shconf [2022-10-26 '
'14:57:13.666161] -l 0 '
'[2022-10-26 14:57:13.666168] '
'--huge-unlink [2022-10-26 '
'14:57:13.666173] '
'--log-level=lib.eal:6 '
'[2022-10-26 14:57:13.666179] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:57:13.666185] '
'--log-level=user1:6 [2022-10-26 '
'14:57:13.666192] --iova-mode=pa '
'[2022-10-26 14:57:13.666199] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:57:13.666206] '
'--match-allocations [2022-10-26 '
'14:57:13.666213] '
'--file-prefix=spdk_pid4723 '
'[2022-10-26 14:57:13.666219] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:57:13.788213] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:57:13.831019] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 14:57:13.831606] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_106.output'),
'state': {'begin': 1666781942.728097,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.124542474746704,
'end': 1666781953.8526394,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_106.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_106.state')},
'cmd_107': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_107.output'),
'state': {'begin': 1666781953.8553784,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7792153358459473,
'end': 1666781954.6345937,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_107.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_107.state')},
'cmd_108': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'390867.35 IOPS 190.85 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'390867.35 IOPS 190.85 MiB/s\n'
'[2022-10-26 14:57:26.619274] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:57:26.619322] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:57:26.619328] bdevperf '
'[2022-10-26 14:57:26.619332] '
'--no-shconf [2022-10-26 '
'14:57:26.619337] -l 0 '
'[2022-10-26 14:57:26.619342] '
'--huge-unlink [2022-10-26 '
'14:57:26.619348] '
'--log-level=lib.eal:6 '
'[2022-10-26 14:57:26.619353] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:57:26.619358] '
'--log-level=user1:6 [2022-10-26 '
'14:57:26.619362] --iova-mode=pa '
'[2022-10-26 14:57:26.619366] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:57:26.619371] '
'--match-allocations [2022-10-26 '
'14:57:26.619376] '
'--file-prefix=spdk_pid4748 '
'[2022-10-26 14:57:26.619380] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:57:26.741441] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:57:26.791036] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 14:57:26.791643] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_108.output'),
'state': {'begin': 1666781955.6668162,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.153231143951416,
'end': 1666781966.8200474,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_108.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_108.state')},
'cmd_109': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_109.output'),
'state': {'begin': 1666781966.8233027,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7791199684143066,
'end': 1666781967.6024227,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_109.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_109.state')},
'cmd_11': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_11.output'),
'state': {'begin': 1666781327.4065323,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8043515682220459,
'end': 1666781328.2108839,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_11.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_11.state')},
'cmd_110': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'404566.90 IOPS 197.54 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'404566.90 IOPS 197.54 MiB/s\n'
'[2022-10-26 14:57:39.566766] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:57:39.566809] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:57:39.566815] bdevperf '
'[2022-10-26 14:57:39.566821] '
'--no-shconf [2022-10-26 '
'14:57:39.566825] -l 0 '
'[2022-10-26 14:57:39.566829] '
'--huge-unlink [2022-10-26 '
'14:57:39.566834] '
'--log-level=lib.eal:6 '
'[2022-10-26 14:57:39.566838] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:57:39.566842] '
'--log-level=user1:6 [2022-10-26 '
'14:57:39.566846] --iova-mode=pa '
'[2022-10-26 14:57:39.566850] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:57:39.566855] '
'--match-allocations [2022-10-26 '
'14:57:39.566859] '
'--file-prefix=spdk_pid4773 '
'[2022-10-26 14:57:39.566863] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:57:39.688845] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:57:39.731022] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 14:57:39.731598] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_110.output'),
'state': {'begin': 1666781968.6418266,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.212659120559692,
'end': 1666781979.8544858,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_110.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_110.state')},
'cmd_111': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_111.output'),
'state': {'begin': 1666781979.857094,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7954108715057373,
'end': 1666781980.652505,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_111.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_111.state')},
'cmd_112': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'404402.85 IOPS 197.46 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'404402.85 IOPS 197.46 MiB/s\n'
'[2022-10-26 14:57:52.662483] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:57:52.662525] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:57:52.662531] bdevperf '
'[2022-10-26 14:57:52.662536] '
'--no-shconf [2022-10-26 '
'14:57:52.662540] -l 0 '
'[2022-10-26 14:57:52.662545] '
'--huge-unlink [2022-10-26 '
'14:57:52.662549] '
'--log-level=lib.eal:6 '
'[2022-10-26 14:57:52.662553] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:57:52.662557] '
'--log-level=user1:6 [2022-10-26 '
'14:57:52.662561] --iova-mode=pa '
'[2022-10-26 14:57:52.662565] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:57:52.662570] '
'--match-allocations [2022-10-26 '
'14:57:52.662574] '
'--file-prefix=spdk_pid4799 '
'[2022-10-26 14:57:52.662579] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:57:52.784566] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:57:52.827141] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 14:57:52.827734] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_112.output'),
'state': {'begin': 1666781981.7609088,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.155568361282349,
'end': 1666781992.9164772,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_112.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_112.state')},
'cmd_113': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_113.output'),
'state': {'begin': 1666781992.9190931,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8321177959442139,
'end': 1666781993.751211,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_113.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_113.state')},
'cmd_114': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'403868.55 IOPS 197.20 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'403868.55 IOPS 197.20 MiB/s\n'
'[2022-10-26 14:58:05.759802] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:58:05.759849] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:58:05.759855] bdevperf '
'[2022-10-26 14:58:05.759860] '
'--no-shconf [2022-10-26 '
'14:58:05.759864] -l 0 '
'[2022-10-26 14:58:05.759869] '
'--huge-unlink [2022-10-26 '
'14:58:05.759874] '
'--log-level=lib.eal:6 '
'[2022-10-26 14:58:05.759878] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:58:05.759883] '
'--log-level=user1:6 [2022-10-26 '
'14:58:05.759887] --iova-mode=pa '
'[2022-10-26 14:58:05.759891] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:58:05.759895] '
'--match-allocations [2022-10-26 '
'14:58:05.759900] '
'--file-prefix=spdk_pid4825 '
'[2022-10-26 14:58:05.759904] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:58:05.881869] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:58:05.919038] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 14:58:05.919619] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_114.output'),
'state': {'begin': 1666781994.8151395,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.180953979492188,
'end': 1666782005.9960935,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_114.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_114.state')},
'cmd_115': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_115.output'),
'state': {'begin': 1666782005.9991379,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.818833589553833,
'end': 1666782006.8179715,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_115.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_115.state')},
'cmd_116': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'382108.42 IOPS 186.58 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'382108.42 IOPS 186.58 MiB/s\n'
'[2022-10-26 14:58:18.828990] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:58:18.829036] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:58:18.829042] bdevperf '
'[2022-10-26 14:58:18.829047] '
'--no-shconf [2022-10-26 '
'14:58:18.829051] -l 0 '
'[2022-10-26 14:58:18.829056] '
'--huge-unlink [2022-10-26 '
'14:58:18.829061] '
'--log-level=lib.eal:6 '
'[2022-10-26 14:58:18.829065] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:58:18.829070] '
'--log-level=user1:6 [2022-10-26 '
'14:58:18.829074] --iova-mode=pa '
'[2022-10-26 14:58:18.829078] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:58:18.829083] '
'--match-allocations [2022-10-26 '
'14:58:18.829088] '
'--file-prefix=spdk_pid4850 '
'[2022-10-26 14:58:18.829092] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:58:18.951199] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:58:18.995101] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 14:58:18.995684] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_116.output'),
'state': {'begin': 1666782007.8945894,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.20103645324707,
'end': 1666782019.0956259,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_116.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_116.state')},
'cmd_117': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_117.output'),
'state': {'begin': 1666782019.098346,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 1.1820051670074463,
'end': 1666782020.2803512,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_117.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_117.state')},
'cmd_118': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'380577.08 IOPS 185.83 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'380577.08 IOPS 185.83 MiB/s\n'
'[2022-10-26 14:58:32.287850] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:58:32.287904] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:58:32.287910] bdevperf '
'[2022-10-26 14:58:32.287915] '
'--no-shconf [2022-10-26 '
'14:58:32.287920] -l 0 '
'[2022-10-26 14:58:32.287924] '
'--huge-unlink [2022-10-26 '
'14:58:32.287929] '
'--log-level=lib.eal:6 '
'[2022-10-26 14:58:32.287933] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:58:32.287937] '
'--log-level=user1:6 [2022-10-26 '
'14:58:32.287942] --iova-mode=pa '
'[2022-10-26 14:58:32.287946] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:58:32.287950] '
'--match-allocations [2022-10-26 '
'14:58:32.287955] '
'--file-prefix=spdk_pid4875 '
'[2022-10-26 14:58:32.287960] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:58:32.409866] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:58:32.450980] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 14:58:32.451586] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_118.output'),
'state': {'begin': 1666782021.3781052,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.193742275238037,
'end': 1666782032.5718474,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_118.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_118.state')},
'cmd_119': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_119.output'),
'state': {'begin': 1666782032.5745347,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8060545921325684,
'end': 1666782033.3805892,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_119.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_119.state')},
'cmd_12': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'137312.00 IOPS 67.05 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'137312.00 IOPS 67.05 MiB/s\n'
'[2022-10-26 14:47:00.239018] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:47:00.239062] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:47:00.239067] bdevperf '
'[2022-10-26 14:47:00.239073] '
'--no-shconf [2022-10-26 '
'14:47:00.239077] -l 0 [2022-10-26 '
'14:47:00.239082] --huge-unlink '
'[2022-10-26 14:47:00.239086] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:47:00.239091] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:47:00.239096] '
'--log-level=user1:6 [2022-10-26 '
'14:47:00.239100] --iova-mode=pa '
'[2022-10-26 14:47:00.239104] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:47:00.239108] '
'--match-allocations [2022-10-26 '
'14:47:00.239112] '
'--file-prefix=spdk_pid3528 '
'[2022-10-26 14:47:00.239117] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:47:00.360863] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:47:00.406981] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:47:00.407550] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_12.output'),
'state': {'begin': 1666781329.3192575,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.100764513015747,
'end': 1666781340.420022,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_12.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_12.state')},
'cmd_120': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'379284.08 IOPS 185.20 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'379284.08 IOPS 185.20 MiB/s\n'
'[2022-10-26 14:58:45.398827] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:58:45.398870] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:58:45.398876] bdevperf '
'[2022-10-26 14:58:45.398881] '
'--no-shconf [2022-10-26 '
'14:58:45.398885] -l 0 '
'[2022-10-26 14:58:45.398890] '
'--huge-unlink [2022-10-26 '
'14:58:45.398895] '
'--log-level=lib.eal:6 '
'[2022-10-26 14:58:45.398899] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:58:45.398904] '
'--log-level=user1:6 [2022-10-26 '
'14:58:45.398908] --iova-mode=pa '
'[2022-10-26 14:58:45.398912] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:58:45.398918] '
'--match-allocations [2022-10-26 '
'14:58:45.398924] '
'--file-prefix=spdk_pid4900 '
'[2022-10-26 14:58:45.398929] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:58:45.521201] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:58:45.571063] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 14:58:45.571640] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_120.output'),
'state': {'begin': 1666782034.445311,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.227607488632202,
'end': 1666782045.6729186,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_120.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_120.state')},
'cmd_13': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_13.output'),
'state': {'begin': 1666781340.4228358,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7960231304168701,
'end': 1666781341.218859,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_13.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_13.state')},
'cmd_14': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'156589.41 IOPS 76.46 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'156589.41 IOPS 76.46 MiB/s\n'
'[2022-10-26 14:47:13.264251] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:47:13.264293] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:47:13.264298] bdevperf '
'[2022-10-26 14:47:13.264303] '
'--no-shconf [2022-10-26 '
'14:47:13.264307] -l 0 [2022-10-26 '
'14:47:13.264311] --huge-unlink '
'[2022-10-26 14:47:13.264316] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:47:13.264320] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:47:13.264324] '
'--log-level=user1:6 [2022-10-26 '
'14:47:13.264328] --iova-mode=pa '
'[2022-10-26 14:47:13.264332] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:47:13.264336] '
'--match-allocations [2022-10-26 '
'14:47:13.264340] '
'--file-prefix=spdk_pid3553 '
'[2022-10-26 14:47:13.264344] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:47:13.386330] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:47:13.430961] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:47:13.431535] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_14.output'),
'state': {'begin': 1666781342.3381026,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.116260051727295,
'end': 1666781353.4543626,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_14.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_14.state')},
'cmd_15': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_15.output'),
'state': {'begin': 1666781353.4572272,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8200883865356445,
'end': 1666781354.2773156,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_15.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_15.state')},
'cmd_16': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'156437.11 IOPS 76.39 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'156437.11 IOPS 76.39 MiB/s\n'
'[2022-10-26 14:47:26.307921] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:47:26.307971] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:47:26.307978] bdevperf '
'[2022-10-26 14:47:26.307985] '
'--no-shconf [2022-10-26 '
'14:47:26.307992] -l 0 [2022-10-26 '
'14:47:26.307997] --huge-unlink '
'[2022-10-26 14:47:26.308003] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:47:26.308009] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:47:26.308016] '
'--log-level=user1:6 [2022-10-26 '
'14:47:26.308022] --iova-mode=pa '
'[2022-10-26 14:47:26.308028] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:47:26.308034] '
'--match-allocations [2022-10-26 '
'14:47:26.308041] '
'--file-prefix=spdk_pid3578 '
'[2022-10-26 14:47:26.308053] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:47:26.429664] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:47:26.462979] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:47:26.463535] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_16.output'),
'state': {'begin': 1666781355.4097142,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.081530332565308,
'end': 1666781366.4912446,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_16.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_16.state')},
'cmd_17': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_17.output'),
'state': {'begin': 1666781366.4939313,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7980461120605469,
'end': 1666781367.2919774,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_17.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_17.state')},
'cmd_18': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'156357.41 IOPS 76.35 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'156357.41 IOPS 76.35 MiB/s\n'
'[2022-10-26 14:47:39.245614] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:47:39.245656] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:47:39.245662] bdevperf '
'[2022-10-26 14:47:39.245667] '
'--no-shconf [2022-10-26 '
'14:47:39.245672] -l 0 [2022-10-26 '
'14:47:39.245676] --huge-unlink '
'[2022-10-26 14:47:39.245681] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:47:39.245685] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:47:39.245689] '
'--log-level=user1:6 [2022-10-26 '
'14:47:39.245694] --iova-mode=pa '
'[2022-10-26 14:47:39.245698] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:47:39.245703] '
'--match-allocations [2022-10-26 '
'14:47:39.245708] '
'--file-prefix=spdk_pid3603 '
'[2022-10-26 14:47:39.245712] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:47:39.367532] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:47:39.411007] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:47:39.411561] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_18.output'),
'state': {'begin': 1666781368.3570135,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.065926790237427,
'end': 1666781379.4229403,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_18.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_18.state')},
'cmd_19': {'output': 'rm: cannot remove '
"'/tmp/spdk_bdev_aio_foo.conf': No "
'such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_19.output'),
'state': {'begin': 1666781379.4257424,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8077173233032227,
'end': 1666781380.2334597,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_19.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_19.state')},
'cmd_20': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'154952.02 IOPS 75.66 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'154952.02 IOPS 75.66 MiB/s\n'
'[2022-10-26 14:47:52.225176] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:47:52.225221] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:47:52.225227] bdevperf '
'[2022-10-26 14:47:52.225232] '
'--no-shconf [2022-10-26 '
'14:47:52.225236] -l 0 [2022-10-26 '
'14:47:52.225240] --huge-unlink '
'[2022-10-26 14:47:52.225245] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:47:52.225249] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:47:52.225253] '
'--log-level=user1:6 [2022-10-26 '
'14:47:52.225257] --iova-mode=pa '
'[2022-10-26 14:47:52.225261] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:47:52.225265] '
'--match-allocations [2022-10-26 '
'14:47:52.225270] '
'--file-prefix=spdk_pid3628 '
'[2022-10-26 14:47:52.225276] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:47:52.346983] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:47:52.398946] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:47:52.399501] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_20.output'),
'state': {'begin': 1666781381.2815309,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.207417964935303,
'end': 1666781392.4889488,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_20.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_20.state')},
'cmd_21': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_21.output'),
'state': {'begin': 1666781392.4910908,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7867584228515625,
'end': 1666781393.2778492,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_21.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_21.state')},
'cmd_22': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'154636.84 IOPS 75.51 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'154636.84 IOPS 75.51 MiB/s\n'
'[2022-10-26 14:48:05.239169] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:48:05.239212] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:48:05.239218] bdevperf '
'[2022-10-26 14:48:05.239223] '
'--no-shconf [2022-10-26 '
'14:48:05.239228] -l 0 [2022-10-26 '
'14:48:05.239233] --huge-unlink '
'[2022-10-26 14:48:05.239237] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:48:05.239241] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:48:05.239246] '
'--log-level=user1:6 [2022-10-26 '
'14:48:05.239250] --iova-mode=pa '
'[2022-10-26 14:48:05.239254] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:48:05.239258] '
'--match-allocations [2022-10-26 '
'14:48:05.239264] '
'--file-prefix=spdk_pid3653 '
'[2022-10-26 14:48:05.239268] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:48:05.361024] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:48:05.410953] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:48:05.411512] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_22.output'),
'state': {'begin': 1666781394.3306637,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.178808212280273,
'end': 1666781405.509472,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_22.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_22.state')},
'cmd_23': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_23.output'),
'state': {'begin': 1666781405.5125322,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7970554828643799,
'end': 1666781406.3095877,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_23.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_23.state')},
'cmd_24': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'154684.51 IOPS 75.53 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'154684.51 IOPS 75.53 MiB/s\n'
'[2022-10-26 14:48:18.287627] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:48:18.287671] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:48:18.287677] bdevperf '
'[2022-10-26 14:48:18.287682] '
'--no-shconf [2022-10-26 '
'14:48:18.287686] -l 0 [2022-10-26 '
'14:48:18.287691] --huge-unlink '
'[2022-10-26 14:48:18.287695] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:48:18.287700] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:48:18.287703] '
'--log-level=user1:6 [2022-10-26 '
'14:48:18.287708] --iova-mode=pa '
'[2022-10-26 14:48:18.287712] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:48:18.287716] '
'--match-allocations [2022-10-26 '
'14:48:18.287721] '
'--file-prefix=spdk_pid3679 '
'[2022-10-26 14:48:18.287727] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:48:18.409256] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:48:18.452516] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:48:18.453072] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_24.output'),
'state': {'begin': 1666781407.3719943,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.219205141067505,
'end': 1666781418.5911994,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_24.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_24.state')},
'cmd_25': {'output': 'rm: cannot remove '
"'/tmp/spdk_bdev_xnvme_libaio.conf': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_25.output'),
'state': {'begin': 1666781418.5938334,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8456614017486572,
'end': 1666781419.4394948,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_25.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_25.state')},
'cmd_26': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'145398.43 IOPS 71.00 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'145398.43 IOPS 71.00 MiB/s\n'
'[2022-10-26 14:48:31.500046] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:48:31.500091] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:48:31.500096] bdevperf '
'[2022-10-26 14:48:31.500102] '
'--no-shconf [2022-10-26 '
'14:48:31.500106] -l 0 [2022-10-26 '
'14:48:31.500111] --huge-unlink '
'[2022-10-26 14:48:31.500115] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:48:31.500119] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:48:31.500123] '
'--log-level=user1:6 [2022-10-26 '
'14:48:31.500128] --iova-mode=pa '
'[2022-10-26 14:48:31.500132] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:48:31.500136] '
'--match-allocations [2022-10-26 '
'14:48:31.500140] '
'--file-prefix=spdk_pid3704 '
'[2022-10-26 14:48:31.500144] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:48:31.621779] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:48:31.662966] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:48:31.663515] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_26.output'),
'state': {'begin': 1666781420.516107,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.241715669631958,
'end': 1666781431.7578228,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_26.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_26.state')},
'cmd_27': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_27.output'),
'state': {'begin': 1666781431.7608914,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.794175386428833,
'end': 1666781432.5550668,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_27.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_27.state')},
'cmd_28': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'146366.04 IOPS 71.47 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'146366.04 IOPS 71.47 MiB/s\n'
'[2022-10-26 14:48:44.543718] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:48:44.543768] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:48:44.543775] bdevperf '
'[2022-10-26 14:48:44.543780] '
'--no-shconf [2022-10-26 '
'14:48:44.543787] -l 0 [2022-10-26 '
'14:48:44.543793] --huge-unlink '
'[2022-10-26 14:48:44.543799] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:48:44.543805] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:48:44.543810] '
'--log-level=user1:6 [2022-10-26 '
'14:48:44.543816] --iova-mode=pa '
'[2022-10-26 14:48:44.543822] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:48:44.543828] '
'--match-allocations [2022-10-26 '
'14:48:44.543834] '
'--file-prefix=spdk_pid3729 '
'[2022-10-26 14:48:44.543839] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:48:44.665842] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:48:44.707021] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:48:44.707577] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_28.output'),
'state': {'begin': 1666781433.6278818,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.179730653762817,
'end': 1666781444.8076124,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_28.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_28.state')},
'cmd_29': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_29.output'),
'state': {'begin': 1666781444.81459,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8071081638336182,
'end': 1666781445.6216981,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_29.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_29.state')},
'cmd_30': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'146618.86 IOPS 71.59 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'146618.86 IOPS 71.59 MiB/s\n'
'[2022-10-26 14:48:57.629719] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:48:57.629762] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:48:57.629768] bdevperf '
'[2022-10-26 14:48:57.629773] '
'--no-shconf [2022-10-26 '
'14:48:57.629778] -l 0 [2022-10-26 '
'14:48:57.629782] --huge-unlink '
'[2022-10-26 14:48:57.629785] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:48:57.629790] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:48:57.629793] '
'--log-level=user1:6 [2022-10-26 '
'14:48:57.629798] --iova-mode=pa '
'[2022-10-26 14:48:57.629801] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:48:57.629805] '
'--match-allocations [2022-10-26 '
'14:48:57.629809] '
'--file-prefix=spdk_pid3755 '
'[2022-10-26 14:48:57.629813] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:48:57.752067] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:48:57.794987] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:48:57.795541] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_30.output'),
'state': {'begin': 1666781446.7268162,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.14713454246521,
'end': 1666781457.8739507,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_30.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_30.state')},
'cmd_31': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_31.output'),
'state': {'begin': 1666781457.875226,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8208591938018799,
'end': 1666781458.6960852,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_31.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_31.state')},
'cmd_32': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'261816.82 IOPS 127.84 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'261816.82 IOPS 127.84 MiB/s\n'
'[2022-10-26 14:49:10.728753] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:49:10.728804] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:49:10.728810] bdevperf '
'[2022-10-26 14:49:10.728814] '
'--no-shconf [2022-10-26 '
'14:49:10.728818] -l 0 [2022-10-26 '
'14:49:10.728823] --huge-unlink '
'[2022-10-26 14:49:10.728827] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:49:10.728831] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:49:10.728835] '
'--log-level=user1:6 [2022-10-26 '
'14:49:10.728839] --iova-mode=pa '
'[2022-10-26 14:49:10.728843] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:49:10.728847] '
'--match-allocations [2022-10-26 '
'14:49:10.728851] '
'--file-prefix=spdk_pid3780 '
'[2022-10-26 14:49:10.728857] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:49:10.850519] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:49:10.894989] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:49:10.895537] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_32.output'),
'state': {'begin': 1666781459.750432,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.17123556137085,
'end': 1666781470.9216676,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_32.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_32.state')},
'cmd_33': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_33.output'),
'state': {'begin': 1666781470.9237893,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.898444414138794,
'end': 1666781471.8222337,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_33.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_33.state')},
'cmd_34': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'261099.59 IOPS 127.49 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'261099.59 IOPS 127.49 MiB/s\n'
'[2022-10-26 14:49:23.852529] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:49:23.852573] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:49:23.852579] bdevperf '
'[2022-10-26 14:49:23.852583] '
'--no-shconf [2022-10-26 '
'14:49:23.852589] -l 0 [2022-10-26 '
'14:49:23.852594] --huge-unlink '
'[2022-10-26 14:49:23.852599] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:49:23.852603] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:49:23.852608] '
'--log-level=user1:6 [2022-10-26 '
'14:49:23.852612] --iova-mode=pa '
'[2022-10-26 14:49:23.852616] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:49:23.852621] '
'--match-allocations [2022-10-26 '
'14:49:23.852626] '
'--file-prefix=spdk_pid3805 '
'[2022-10-26 14:49:23.852632] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:49:23.974359] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:49:24.027041] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:49:24.027602] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_34.output'),
'state': {'begin': 1666781472.9142237,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.13983416557312,
'end': 1666781484.0540578,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_34.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_34.state')},
'cmd_35': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_35.output'),
'state': {'begin': 1666781484.0572314,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8143582344055176,
'end': 1666781484.8715897,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_35.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_35.state')},
'cmd_36': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'260889.10 IOPS 127.39 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'260889.10 IOPS 127.39 MiB/s\n'
'[2022-10-26 14:49:36.826143] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:49:36.826185] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:49:36.826191] bdevperf '
'[2022-10-26 14:49:36.826195] '
'--no-shconf [2022-10-26 '
'14:49:36.826200] -l 0 [2022-10-26 '
'14:49:36.826204] --huge-unlink '
'[2022-10-26 14:49:36.826209] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:49:36.826213] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:49:36.826218] '
'--log-level=user1:6 [2022-10-26 '
'14:49:36.826222] --iova-mode=pa '
'[2022-10-26 14:49:36.826226] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:49:36.826230] '
'--match-allocations [2022-10-26 '
'14:49:36.826235] '
'--file-prefix=spdk_pid3830 '
'[2022-10-26 14:49:36.826239] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:49:36.947892] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:49:36.991044] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:49:36.991602] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_36.output'),
'state': {'begin': 1666781485.9342191,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.135831117630005,
'end': 1666781497.0700502,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_36.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_36.state')},
'cmd_37': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_37.output'),
'state': {'begin': 1666781497.072821,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8443262577056885,
'end': 1666781497.9171472,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_37.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_37.state')},
'cmd_38': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'230826.32 IOPS 112.71 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'230826.32 IOPS 112.71 MiB/s\n'
'[2022-10-26 14:49:49.866229] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:49:49.866273] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:49:49.866279] bdevperf '
'[2022-10-26 14:49:49.866283] '
'--no-shconf [2022-10-26 '
'14:49:49.866288] -l 0 [2022-10-26 '
'14:49:49.866293] --huge-unlink '
'[2022-10-26 14:49:49.866298] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:49:49.866303] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:49:49.866307] '
'--log-level=user1:6 [2022-10-26 '
'14:49:49.866311] --iova-mode=pa '
'[2022-10-26 14:49:49.866316] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:49:49.866321] '
'--match-allocations [2022-10-26 '
'14:49:49.866325] '
'--file-prefix=spdk_pid3855 '
'[2022-10-26 14:49:49.866330] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:49:49.987927] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:49:50.031010] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:49:50.031564] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_38.output'),
'state': {'begin': 1666781498.9793825,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.064125537872314,
'end': 1666781510.043508,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_38.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_38.state')},
'cmd_39': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_39.output'),
'state': {'begin': 1666781510.046105,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7939174175262451,
'end': 1666781510.8400223,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_39.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_39.state')},
'cmd_40': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'229739.44 IOPS 112.18 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'229739.44 IOPS 112.18 MiB/s\n'
'[2022-10-26 14:50:02.780961] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:50:02.781004] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:50:02.781010] bdevperf '
'[2022-10-26 14:50:02.781015] '
'--no-shconf [2022-10-26 '
'14:50:02.781020] -l 0 [2022-10-26 '
'14:50:02.781026] --huge-unlink '
'[2022-10-26 14:50:02.781030] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:50:02.781034] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:50:02.781038] '
'--log-level=user1:6 [2022-10-26 '
'14:50:02.781042] --iova-mode=pa '
'[2022-10-26 14:50:02.781046] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:50:02.781050] '
'--match-allocations [2022-10-26 '
'14:50:02.781060] '
'--file-prefix=spdk_pid3880 '
'[2022-10-26 14:50:02.781064] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:50:02.902700] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:50:02.946942] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:50:02.947497] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_40.output'),
'state': {'begin': 1666781511.8811512,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.078004837036133,
'end': 1666781522.959156,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_40.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_40.state')},
'cmd_41': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_41.output'),
'state': {'begin': 1666781522.9622078,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8215856552124023,
'end': 1666781523.7837934,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_41.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_41.state')},
'cmd_42': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'231226.96 IOPS 112.90 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'231226.96 IOPS 112.90 MiB/s\n'
'[2022-10-26 14:50:15.813398] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:50:15.813439] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:50:15.813445] bdevperf '
'[2022-10-26 14:50:15.813450] '
'--no-shconf [2022-10-26 '
'14:50:15.813455] -l 0 [2022-10-26 '
'14:50:15.813460] --huge-unlink '
'[2022-10-26 14:50:15.813464] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:50:15.813470] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:50:15.813474] '
'--log-level=user1:6 [2022-10-26 '
'14:50:15.813478] --iova-mode=pa '
'[2022-10-26 14:50:15.813482] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:50:15.813487] '
'--match-allocations [2022-10-26 '
'14:50:15.813491] '
'--file-prefix=spdk_pid3906 '
'[2022-10-26 14:50:15.813495] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:50:15.935094] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:50:15.978913] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:50:15.979453] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_42.output'),
'state': {'begin': 1666781524.9029205,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.087957382202148,
'end': 1666781535.9908779,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_42.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_42.state')},
'cmd_43': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_43.output'),
'state': {'begin': 1666781535.9935784,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8295695781707764,
'end': 1666781536.823148,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_43.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_43.state')},
'cmd_44': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'282909.29 IOPS 138.14 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'282909.29 IOPS 138.14 MiB/s\n'
'[2022-10-26 14:50:28.780387] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:50:28.780434] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:50:28.780440] bdevperf '
'[2022-10-26 14:50:28.780445] '
'--no-shconf [2022-10-26 '
'14:50:28.780449] -l 0 [2022-10-26 '
'14:50:28.780453] --huge-unlink '
'[2022-10-26 14:50:28.780458] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:50:28.780463] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:50:28.780467] '
'--log-level=user1:6 [2022-10-26 '
'14:50:28.780471] --iova-mode=pa '
'[2022-10-26 14:50:28.780475] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:50:28.780479] '
'--match-allocations [2022-10-26 '
'14:50:28.780485] '
'--file-prefix=spdk_pid3931 '
'[2022-10-26 14:50:28.780491] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:50:28.902341] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:50:28.948483] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:50:28.949036] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_44.output'),
'state': {'begin': 1666781537.8824975,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.088404178619385,
'end': 1666781548.9709017,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_44.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_44.state')},
'cmd_45': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_45.output'),
'state': {'begin': 1666781548.9792178,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.800908088684082,
'end': 1666781549.7801259,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_45.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_45.state')},
'cmd_46': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'283679.60 IOPS 138.52 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'283679.60 IOPS 138.52 MiB/s\n'
'[2022-10-26 14:50:41.753061] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:50:41.753108] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:50:41.753113] bdevperf '
'[2022-10-26 14:50:41.753119] '
'--no-shconf [2022-10-26 '
'14:50:41.753123] -l 0 [2022-10-26 '
'14:50:41.753128] --huge-unlink '
'[2022-10-26 14:50:41.753132] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:50:41.753138] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:50:41.753142] '
'--log-level=user1:6 [2022-10-26 '
'14:50:41.753146] --iova-mode=pa '
'[2022-10-26 14:50:41.753150] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:50:41.753154] '
'--match-allocations [2022-10-26 '
'14:50:41.753160] '
'--file-prefix=spdk_pid3957 '
'[2022-10-26 14:50:41.753164] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:50:41.874953] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:50:41.915040] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:50:41.915590] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_46.output'),
'state': {'begin': 1666781550.867793,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.061692476272583,
'end': 1666781561.9294856,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_46.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_46.state')},
'cmd_47': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_47.output'),
'state': {'begin': 1666781561.9321043,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7853102684020996,
'end': 1666781562.7174146,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_47.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_47.state')},
'cmd_48': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'283155.55 IOPS 138.26 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'283155.55 IOPS 138.26 MiB/s\n'
'[2022-10-26 14:50:54.724079] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:50:54.724121] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:50:54.724127] bdevperf '
'[2022-10-26 14:50:54.724132] '
'--no-shconf [2022-10-26 '
'14:50:54.724137] -l 0 [2022-10-26 '
'14:50:54.724141] --huge-unlink '
'[2022-10-26 14:50:54.724146] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:50:54.724149] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:50:54.724154] '
'--log-level=user1:6 [2022-10-26 '
'14:50:54.724157] --iova-mode=pa '
'[2022-10-26 14:50:54.724161] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:50:54.724165] '
'--match-allocations [2022-10-26 '
'14:50:54.724169] '
'--file-prefix=spdk_pid3984 '
'[2022-10-26 14:50:54.724175] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:50:54.845795] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:50:54.887066] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:50:54.887623] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_48.output'),
'state': {'begin': 1666781563.814571,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.088241577148438,
'end': 1666781574.9028125,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_48.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_48.state')},
'cmd_49': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_49.output'),
'state': {'begin': 1666781574.905429,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8327929973602295,
'end': 1666781575.738222,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_49.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_49.state')},
'cmd_50': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'281466.49 IOPS 137.43 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'281466.49 IOPS 137.43 MiB/s\n'
'[2022-10-26 14:51:07.680050] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:51:07.680094] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:51:07.680101] bdevperf '
'[2022-10-26 14:51:07.680106] '
'--no-shconf [2022-10-26 '
'14:51:07.680111] -l 0 [2022-10-26 '
'14:51:07.680118] --huge-unlink '
'[2022-10-26 14:51:07.680126] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:51:07.680132] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:51:07.680137] '
'--log-level=user1:6 [2022-10-26 '
'14:51:07.680142] --iova-mode=pa '
'[2022-10-26 14:51:07.680149] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:51:07.680154] '
'--match-allocations [2022-10-26 '
'14:51:07.680160] '
'--file-prefix=spdk_pid4010 '
'[2022-10-26 14:51:07.680167] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:51:07.801780] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:51:07.839036] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:51:07.839594] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_50.output'),
'state': {'begin': 1666781576.7971725,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.1195068359375,
'end': 1666781587.9166794,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_50.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_50.state')},
'cmd_51': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_51.output'),
'state': {'begin': 1666781587.9197438,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7945308685302734,
'end': 1666781588.7142746,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_51.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_51.state')},
'cmd_52': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'280747.16 IOPS 137.08 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'280747.16 IOPS 137.08 MiB/s\n'
'[2022-10-26 14:51:20.723914] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:51:20.723958] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:51:20.723964] bdevperf '
'[2022-10-26 14:51:20.723968] '
'--no-shconf [2022-10-26 '
'14:51:20.723973] -l 0 [2022-10-26 '
'14:51:20.723979] --huge-unlink '
'[2022-10-26 14:51:20.723983] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:51:20.723988] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:51:20.723992] '
'--log-level=user1:6 [2022-10-26 '
'14:51:20.723997] --iova-mode=pa '
'[2022-10-26 14:51:20.724001] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:51:20.724004] '
'--match-allocations [2022-10-26 '
'14:51:20.724009] '
'--file-prefix=spdk_pid4036 '
'[2022-10-26 14:51:20.724014] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:51:20.845500] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:51:20.895019] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:51:20.895578] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_52.output'),
'state': {'begin': 1666781589.80861,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.155845403671265,
'end': 1666781600.9644554,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_52.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_52.state')},
'cmd_53': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_53.output'),
'state': {'begin': 1666781600.9671118,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7982139587402344,
'end': 1666781601.7653258,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_53.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_53.state')},
'cmd_54': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'280777.96 IOPS 137.10 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'280777.96 IOPS 137.10 MiB/s\n'
'[2022-10-26 14:51:33.686533] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:51:33.686578] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:51:33.686584] bdevperf '
'[2022-10-26 14:51:33.686589] '
'--no-shconf [2022-10-26 '
'14:51:33.686594] -l 0 [2022-10-26 '
'14:51:33.686598] --huge-unlink '
'[2022-10-26 14:51:33.686603] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:51:33.686608] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:51:33.686612] '
'--log-level=user1:6 [2022-10-26 '
'14:51:33.686616] --iova-mode=pa '
'[2022-10-26 14:51:33.686620] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:51:33.686625] '
'--match-allocations [2022-10-26 '
'14:51:33.686630] '
'--file-prefix=spdk_pid4063 '
'[2022-10-26 14:51:33.686635] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:51:33.808399] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:51:33.851003] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:51:33.851560] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_54.output'),
'state': {'begin': 1666781602.806074,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.11859655380249,
'end': 1666781613.9246705,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_54.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_54.state')},
'cmd_55': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_55.output'),
'state': {'begin': 1666781613.9272351,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8031032085418701,
'end': 1666781614.7303383,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_55.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_55.state')},
'cmd_56': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'267085.41 IOPS 130.41 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'267085.41 IOPS 130.41 MiB/s\n'
'[2022-10-26 14:51:46.743969] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:51:46.744011] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:51:46.744017] bdevperf '
'[2022-10-26 14:51:46.744021] '
'--no-shconf [2022-10-26 '
'14:51:46.744026] -l 0 [2022-10-26 '
'14:51:46.744030] --huge-unlink '
'[2022-10-26 14:51:46.744034] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:51:46.744038] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:51:46.744044] '
'--log-level=user1:6 [2022-10-26 '
'14:51:46.744048] --iova-mode=pa '
'[2022-10-26 14:51:46.744053] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:51:46.744058] '
'--match-allocations [2022-10-26 '
'14:51:46.744064] '
'--file-prefix=spdk_pid4090 '
'[2022-10-26 14:51:46.744068] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:51:46.866027] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:51:46.903064] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:51:46.903629] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_56.output'),
'state': {'begin': 1666781615.849239,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.135571241378784,
'end': 1666781626.9848104,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_56.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_56.state')},
'cmd_57': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_57.output'),
'state': {'begin': 1666781626.9873095,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8279106616973877,
'end': 1666781627.81522,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_57.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_57.state')},
'cmd_58': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'267177.11 IOPS 130.46 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'267177.11 IOPS 130.46 MiB/s\n'
'[2022-10-26 14:51:59.778212] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:51:59.778256] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:51:59.778262] bdevperf '
'[2022-10-26 14:51:59.778267] '
'--no-shconf [2022-10-26 '
'14:51:59.778271] -l 0 [2022-10-26 '
'14:51:59.778276] --huge-unlink '
'[2022-10-26 14:51:59.778281] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:51:59.778285] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:51:59.778291] '
'--log-level=user1:6 [2022-10-26 '
'14:51:59.778295] --iova-mode=pa '
'[2022-10-26 14:51:59.778299] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:51:59.778303] '
'--match-allocations [2022-10-26 '
'14:51:59.778307] '
'--file-prefix=spdk_pid4115 '
'[2022-10-26 14:51:59.778313] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:51:59.900002] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:51:59.951001] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:51:59.951562] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_58.output'),
'state': {'begin': 1666781628.8598037,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.180998086929321,
'end': 1666781640.0408018,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_58.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_58.state')},
'cmd_59': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_59.output'),
'state': {'begin': 1666781640.04412,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8200516700744629,
'end': 1666781640.8641717,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_59.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_59.state')},
'cmd_60': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'266611.68 IOPS 130.18 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'266611.68 IOPS 130.18 MiB/s\n'
'[2022-10-26 14:52:12.820906] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:52:12.820957] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:52:12.820965] bdevperf '
'[2022-10-26 14:52:12.820972] '
'--no-shconf [2022-10-26 '
'14:52:12.820978] -l 0 [2022-10-26 '
'14:52:12.820984] --huge-unlink '
'[2022-10-26 14:52:12.820990] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:52:12.820997] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:52:12.821003] '
'--log-level=user1:6 [2022-10-26 '
'14:52:12.821011] --iova-mode=pa '
'[2022-10-26 14:52:12.821019] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:52:12.821025] '
'--match-allocations [2022-10-26 '
'14:52:12.821032] '
'--file-prefix=spdk_pid4140 '
'[2022-10-26 14:52:12.821039] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:52:12.942842] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:52:12.987011] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:52:12.987571] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_60.output'),
'state': {'begin': 1666781641.9165711,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.18920350074768,
'end': 1666781653.1057746,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_60.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_60.state')},
'cmd_61': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_61.output'),
'state': {'begin': 1666781653.1083791,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8270316123962402,
'end': 1666781653.9354107,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_61.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_61.state')},
'cmd_62': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'409829.09 IOPS 200.11 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'409829.09 IOPS 200.11 MiB/s\n'
'[2022-10-26 14:52:25.997001] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:52:25.997050] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:52:25.997056] bdevperf '
'[2022-10-26 14:52:25.997060] '
'--no-shconf [2022-10-26 '
'14:52:25.997064] -l 0 [2022-10-26 '
'14:52:25.997069] --huge-unlink '
'[2022-10-26 14:52:25.997074] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:52:25.997078] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:52:25.997083] '
'--log-level=user1:6 [2022-10-26 '
'14:52:25.997087] --iova-mode=pa '
'[2022-10-26 14:52:25.997091] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:52:25.997095] '
'--match-allocations [2022-10-26 '
'14:52:25.997100] '
'--file-prefix=spdk_pid4165 '
'[2022-10-26 14:52:25.997105] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:52:26.118822] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:52:26.162992] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:52:26.163553] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_62.output'),
'state': {'begin': 1666781655.1041815,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.088993072509766,
'end': 1666781666.1931746,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_62.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_62.state')},
'cmd_63': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_63.output'),
'state': {'begin': 1666781666.1958718,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8419077396392822,
'end': 1666781667.0377796,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_63.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_63.state')},
'cmd_64': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'453036.84 IOPS 221.21 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'453036.84 IOPS 221.21 MiB/s\n'
'[2022-10-26 14:52:39.039669] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:52:39.039712] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:52:39.039717] bdevperf '
'[2022-10-26 14:52:39.039722] '
'--no-shconf [2022-10-26 '
'14:52:39.039726] -l 0 [2022-10-26 '
'14:52:39.039731] --huge-unlink '
'[2022-10-26 14:52:39.039736] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:52:39.039740] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:52:39.039745] '
'--log-level=user1:6 [2022-10-26 '
'14:52:39.039749] --iova-mode=pa '
'[2022-10-26 14:52:39.039752] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:52:39.039757] '
'--match-allocations [2022-10-26 '
'14:52:39.039761] '
'--file-prefix=spdk_pid4190 '
'[2022-10-26 14:52:39.039766] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:52:39.161430] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:52:39.203055] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:52:39.203611] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_64.output'),
'state': {'begin': 1666781668.1494107,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.081124782562256,
'end': 1666781679.2305355,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_64.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_64.state')},
'cmd_65': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_65.output'),
'state': {'begin': 1666781679.2331562,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 1.0669896602630615,
'end': 1666781680.3001459,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_65.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_65.state')},
'cmd_66': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'493618.05 IOPS 241.02 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'493618.05 IOPS 241.02 MiB/s\n'
'[2022-10-26 14:52:52.548321] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:52:52.548367] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:52:52.548372] bdevperf '
'[2022-10-26 14:52:52.548377] '
'--no-shconf [2022-10-26 '
'14:52:52.548381] -l 0 [2022-10-26 '
'14:52:52.548385] --huge-unlink '
'[2022-10-26 14:52:52.548390] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:52:52.548394] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:52:52.548398] '
'--log-level=user1:6 [2022-10-26 '
'14:52:52.548402] --iova-mode=pa '
'[2022-10-26 14:52:52.548406] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:52:52.548410] '
'--match-allocations [2022-10-26 '
'14:52:52.548415] '
'--file-prefix=spdk_pid4215 '
'[2022-10-26 14:52:52.548422] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:52:52.669991] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:52:52.711005] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:52:52.711571] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_66.output'),
'state': {'begin': 1666781681.5408564,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.20149278640747,
'end': 1666781692.7423491,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_66.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_66.state')},
'cmd_67': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_67.output'),
'state': {'begin': 1666781692.745624,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7977368831634521,
'end': 1666781693.543361,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_67.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_67.state')},
'cmd_68': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'284288.83 IOPS 138.81 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'284288.83 IOPS 138.81 MiB/s\n'
'[2022-10-26 14:53:05.541424] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:53:05.541465] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:53:05.541471] bdevperf '
'[2022-10-26 14:53:05.541475] '
'--no-shconf [2022-10-26 '
'14:53:05.541480] -l 0 [2022-10-26 '
'14:53:05.541484] --huge-unlink '
'[2022-10-26 14:53:05.541488] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:53:05.541493] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:53:05.541497] '
'--log-level=user1:6 [2022-10-26 '
'14:53:05.541501] --iova-mode=pa '
'[2022-10-26 14:53:05.541506] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:53:05.541511] '
'--match-allocations [2022-10-26 '
'14:53:05.541515] '
'--file-prefix=spdk_pid4240 '
'[2022-10-26 14:53:05.541520] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:53:05.663378] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:53:05.706987] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:53:05.707569] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_68.output'),
'state': {'begin': 1666781694.647939,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.072965621948242,
'end': 1666781705.7209046,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_68.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_68.state')},
'cmd_69': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_69.output'),
'state': {'begin': 1666781705.7281647,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7996237277984619,
'end': 1666781706.5277884,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_69.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_69.state')},
'cmd_70': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'283826.39 IOPS 138.59 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'283826.39 IOPS 138.59 MiB/s\n'
'[2022-10-26 14:53:18.550947] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:53:18.550994] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:53:18.551000] bdevperf '
'[2022-10-26 14:53:18.551004] '
'--no-shconf [2022-10-26 '
'14:53:18.551009] -l 0 [2022-10-26 '
'14:53:18.551013] --huge-unlink '
'[2022-10-26 14:53:18.551018] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:53:18.551022] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:53:18.551027] '
'--log-level=user1:6 [2022-10-26 '
'14:53:18.551031] --iova-mode=pa '
'[2022-10-26 14:53:18.551035] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:53:18.551039] '
'--match-allocations [2022-10-26 '
'14:53:18.551043] '
'--file-prefix=spdk_pid4266 '
'[2022-10-26 14:53:18.551048] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:53:18.672613] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:53:18.707088] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:53:18.707639] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_70.output'),
'state': {'begin': 1666781707.6440644,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.078567504882812,
'end': 1666781718.722632,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_70.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_70.state')},
'cmd_71': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_71.output'),
'state': {'begin': 1666781718.7253253,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8233606815338135,
'end': 1666781719.548686,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_71.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_71.state')},
'cmd_72': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'283493.26 IOPS 138.42 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'283493.26 IOPS 138.42 MiB/s\n'
'[2022-10-26 14:53:31.638209] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:53:31.638256] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:53:31.638262] bdevperf '
'[2022-10-26 14:53:31.638266] '
'--no-shconf [2022-10-26 '
'14:53:31.638271] -l 0 [2022-10-26 '
'14:53:31.638276] --huge-unlink '
'[2022-10-26 14:53:31.638280] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:53:31.638286] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:53:31.638290] '
'--log-level=user1:6 [2022-10-26 '
'14:53:31.638294] --iova-mode=pa '
'[2022-10-26 14:53:31.638299] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:53:31.638304] '
'--match-allocations [2022-10-26 '
'14:53:31.638308] '
'--file-prefix=spdk_pid4291 '
'[2022-10-26 14:53:31.638313] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:53:31.760182] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:53:31.803079] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:53:31.803648] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_72.output'),
'state': {'begin': 1666781720.663,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.16416072845459,
'end': 1666781731.8271608,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_72.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_72.state')},
'cmd_73': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_73.output'),
'state': {'begin': 1666781731.829704,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8839006423950195,
'end': 1666781732.7136047,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_73.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_73.state')},
'cmd_74': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'364445.67 IOPS 177.95 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'364445.67 IOPS 177.95 MiB/s\n'
'[2022-10-26 14:53:44.673297] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:53:44.673344] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:53:44.673349] bdevperf '
'[2022-10-26 14:53:44.673354] '
'--no-shconf [2022-10-26 '
'14:53:44.673358] -l 0 [2022-10-26 '
'14:53:44.673363] --huge-unlink '
'[2022-10-26 14:53:44.673368] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:53:44.673373] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:53:44.673377] '
'--log-level=user1:6 [2022-10-26 '
'14:53:44.673381] --iova-mode=pa '
'[2022-10-26 14:53:44.673386] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:53:44.673392] '
'--match-allocations [2022-10-26 '
'14:53:44.673396] '
'--file-prefix=spdk_pid4316 '
'[2022-10-26 14:53:44.673401] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:53:44.795838] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:53:44.839049] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:53:44.839650] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_74.output'),
'state': {'begin': 1666781733.7894955,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.076178550720215,
'end': 1666781744.865674,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_74.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_74.state')},
'cmd_75': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_75.output'),
'state': {'begin': 1666781744.8684454,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8356173038482666,
'end': 1666781745.7040627,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_75.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_75.state')},
'cmd_76': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'366439.44 IOPS 178.93 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'366439.44 IOPS 178.93 MiB/s\n'
'[2022-10-26 14:53:57.645285] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:53:57.645331] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:53:57.645337] bdevperf '
'[2022-10-26 14:53:57.645342] '
'--no-shconf [2022-10-26 '
'14:53:57.645347] -l 0 [2022-10-26 '
'14:53:57.645352] --huge-unlink '
'[2022-10-26 14:53:57.645356] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:53:57.645361] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:53:57.645366] '
'--log-level=user1:6 [2022-10-26 '
'14:53:57.645371] --iova-mode=pa '
'[2022-10-26 14:53:57.645376] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:53:57.645381] '
'--match-allocations [2022-10-26 '
'14:53:57.645385] '
'--file-prefix=spdk_pid4342 '
'[2022-10-26 14:53:57.645390] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:53:57.767573] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:53:57.807080] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:53:57.807678] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_76.output'),
'state': {'begin': 1666781746.7566874,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.076358556747437,
'end': 1666781757.833046,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_76.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_76.state')},
'cmd_77': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_77.output'),
'state': {'begin': 1666781757.8356624,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8008663654327393,
'end': 1666781758.6365287,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_77.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_77.state')},
'cmd_78': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'363187.05 IOPS 177.34 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'363187.05 IOPS 177.34 MiB/s\n'
'[2022-10-26 14:54:10.638786] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:54:10.638833] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:54:10.638840] bdevperf '
'[2022-10-26 14:54:10.638845] '
'--no-shconf [2022-10-26 '
'14:54:10.638850] -l 0 [2022-10-26 '
'14:54:10.638854] --huge-unlink '
'[2022-10-26 14:54:10.638858] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:54:10.638862] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:54:10.638866] '
'--log-level=user1:6 [2022-10-26 '
'14:54:10.638871] --iova-mode=pa '
'[2022-10-26 14:54:10.638876] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:54:10.638880] '
'--match-allocations [2022-10-26 '
'14:54:10.638884] '
'--file-prefix=spdk_pid4368 '
'[2022-10-26 14:54:10.638888] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:54:10.760857] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:54:10.799163] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:54:10.799767] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_78.output'),
'state': {'begin': 1666781759.7290328,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.097384691238403,
'end': 1666781770.8264174,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_78.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_78.state')},
'cmd_79': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_79.output'),
'state': {'begin': 1666781770.8294969,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8280513286590576,
'end': 1666781771.6575482,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_79.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_79.state')},
'cmd_80': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'384959.50 IOPS 187.97 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'384959.50 IOPS 187.97 MiB/s\n'
'[2022-10-26 14:54:23.653940] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:54:23.653988] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:54:23.653994] bdevperf '
'[2022-10-26 14:54:23.653998] '
'--no-shconf [2022-10-26 '
'14:54:23.654003] -l 0 [2022-10-26 '
'14:54:23.654007] --huge-unlink '
'[2022-10-26 14:54:23.654012] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:54:23.654017] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:54:23.654022] '
'--log-level=user1:6 [2022-10-26 '
'14:54:23.654026] --iova-mode=pa '
'[2022-10-26 14:54:23.654030] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:54:23.654035] '
'--match-allocations [2022-10-26 '
'14:54:23.654040] '
'--file-prefix=spdk_pid4393 '
'[2022-10-26 14:54:23.654045] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:54:23.776052] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:54:23.828570] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:54:23.829162] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_80.output'),
'state': {'begin': 1666781772.7136788,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.202033996582031,
'end': 1666781783.9157128,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_80.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_80.state')},
'cmd_81': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_81.output'),
'state': {'begin': 1666781783.9179158,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8477661609649658,
'end': 1666781784.765682,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_81.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_81.state')},
'cmd_82': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'384611.58 IOPS 187.80 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'384611.58 IOPS 187.80 MiB/s\n'
'[2022-10-26 14:54:36.787399] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:54:36.787459] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:54:36.787467] bdevperf '
'[2022-10-26 14:54:36.787474] '
'--no-shconf [2022-10-26 '
'14:54:36.787481] -l 0 [2022-10-26 '
'14:54:36.787486] --huge-unlink '
'[2022-10-26 14:54:36.787491] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:54:36.787497] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:54:36.787503] '
'--log-level=user1:6 [2022-10-26 '
'14:54:36.787510] --iova-mode=pa '
'[2022-10-26 14:54:36.787516] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:54:36.787523] '
'--match-allocations [2022-10-26 '
'14:54:36.787530] '
'--file-prefix=spdk_pid4418 '
'[2022-10-26 14:54:36.787537] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:54:36.909401] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:54:36.951018] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:54:36.951625] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_82.output'),
'state': {'begin': 1666781785.8780043,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.161867618560791,
'end': 1666781797.039872,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_82.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_82.state')},
'cmd_83': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_83.output'),
'state': {'begin': 1666781797.0430489,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8415968418121338,
'end': 1666781797.8846457,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_83.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_83.state')},
'cmd_84': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'387768.06 IOPS 189.34 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'387768.06 IOPS 189.34 MiB/s\n'
'[2022-10-26 14:54:49.793371] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:54:49.793413] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:54:49.793420] bdevperf '
'[2022-10-26 14:54:49.793424] '
'--no-shconf [2022-10-26 '
'14:54:49.793429] -l 0 [2022-10-26 '
'14:54:49.793434] --huge-unlink '
'[2022-10-26 14:54:49.793438] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:54:49.793444] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:54:49.793448] '
'--log-level=user1:6 [2022-10-26 '
'14:54:49.793452] --iova-mode=pa '
'[2022-10-26 14:54:49.793456] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:54:49.793460] '
'--match-allocations [2022-10-26 '
'14:54:49.793466] '
'--file-prefix=spdk_pid4444 '
'[2022-10-26 14:54:49.793470] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:54:49.915385] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:54:49.955076] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:54:49.955675] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_84.output'),
'state': {'begin': 1666781798.9065425,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.138028144836426,
'end': 1666781810.0445707,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_84.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_84.state')},
'cmd_85': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_85.output'),
'state': {'begin': 1666781810.0475254,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8003859519958496,
'end': 1666781810.8479114,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_85.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_85.state')},
'cmd_86': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'336723.59 IOPS 164.42 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'336723.59 IOPS 164.42 MiB/s\n'
'[2022-10-26 14:55:02.826118] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:55:02.826165] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:55:02.826170] bdevperf '
'[2022-10-26 14:55:02.826176] '
'--no-shconf [2022-10-26 '
'14:55:02.826180] -l 0 [2022-10-26 '
'14:55:02.826185] --huge-unlink '
'[2022-10-26 14:55:02.826189] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:55:02.826194] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:55:02.826198] '
'--log-level=user1:6 [2022-10-26 '
'14:55:02.826202] --iova-mode=pa '
'[2022-10-26 14:55:02.826206] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:55:02.826210] '
'--match-allocations [2022-10-26 '
'14:55:02.826215] '
'--file-prefix=spdk_pid4469 '
'[2022-10-26 14:55:02.826220] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:55:02.948254] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:55:02.990987] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:55:02.991582] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_86.output'),
'state': {'begin': 1666781811.881793,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.215790033340454,
'end': 1666781823.097583,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_86.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_86.state')},
'cmd_87': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_87.output'),
'state': {'begin': 1666781823.1001909,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8029780387878418,
'end': 1666781823.903169,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_87.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_87.state')},
'cmd_88': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'338676.43 IOPS 165.37 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'338676.43 IOPS 165.37 MiB/s\n'
'[2022-10-26 14:55:15.880904] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:55:15.880951] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:55:15.880957] bdevperf '
'[2022-10-26 14:55:15.880962] '
'--no-shconf [2022-10-26 '
'14:55:15.880966] -l 0 [2022-10-26 '
'14:55:15.880971] --huge-unlink '
'[2022-10-26 14:55:15.880976] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:55:15.880981] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:55:15.880985] '
'--log-level=user1:6 [2022-10-26 '
'14:55:15.880989] --iova-mode=pa '
'[2022-10-26 14:55:15.880995] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:55:15.881000] '
'--match-allocations [2022-10-26 '
'14:55:15.881004] '
'--file-prefix=spdk_pid4494 '
'[2022-10-26 14:55:15.881009] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:55:16.003039] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:55:16.042971] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:55:16.043568] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_88.output'),
'state': {'begin': 1666781824.9668732,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.181619882583618,
'end': 1666781836.148493,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_88.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_88.state')},
'cmd_89': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_89.output'),
'state': {'begin': 1666781836.1511254,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8683156967163086,
'end': 1666781837.0194411,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_89.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_89.state')},
'cmd_90': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'337224.20 IOPS 164.66 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'337224.20 IOPS 164.66 MiB/s\n'
'[2022-10-26 14:55:29.004977] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:55:29.005022] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:55:29.005028] bdevperf '
'[2022-10-26 14:55:29.005033] '
'--no-shconf [2022-10-26 '
'14:55:29.005037] -l 0 [2022-10-26 '
'14:55:29.005042] --huge-unlink '
'[2022-10-26 14:55:29.005046] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:55:29.005051] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:55:29.005055] '
'--log-level=user1:6 [2022-10-26 '
'14:55:29.005060] --iova-mode=pa '
'[2022-10-26 14:55:29.005064] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:55:29.005068] '
'--match-allocations [2022-10-26 '
'14:55:29.005072] '
'--file-prefix=spdk_pid4520 '
'[2022-10-26 14:55:29.005077] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:55:29.127554] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:55:29.171155] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:55:29.171762] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_90.output'),
'state': {'begin': 1666781838.11049,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.155629873275757,
'end': 1666781849.26612,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_90.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_90.state')},
'cmd_91': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_91.output'),
'state': {'begin': 1666781849.2691355,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8817453384399414,
'end': 1666781850.1508808,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_91.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_91.state')},
'cmd_92': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'559323.89 IOPS 273.11 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'559323.89 IOPS 273.11 MiB/s\n'
'[2022-10-26 14:55:42.157720] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:55:42.157764] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:55:42.157769] bdevperf '
'[2022-10-26 14:55:42.157774] '
'--no-shconf [2022-10-26 '
'14:55:42.157779] -l 0 [2022-10-26 '
'14:55:42.157784] --huge-unlink '
'[2022-10-26 14:55:42.157788] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:55:42.157792] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:55:42.157795] '
'--log-level=user1:6 [2022-10-26 '
'14:55:42.157800] --iova-mode=pa '
'[2022-10-26 14:55:42.157804] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:55:42.157809] '
'--match-allocations [2022-10-26 '
'14:55:42.157813] '
'--file-prefix=spdk_pid4546 '
'[2022-10-26 14:55:42.157818] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:55:42.279857] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:55:42.323115] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:55:42.323737] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_92.output'),
'state': {'begin': 1666781851.218421,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 8 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.138697147369385,
'end': 1666781862.3571181,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_92.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_92.state')},
'cmd_93': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_93.output'),
'state': {'begin': 1666781862.3601584,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 1.1332049369812012,
'end': 1666781863.4933634,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_93.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_93.state')},
'cmd_94': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'629634.72 IOPS 307.44 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'629634.72 IOPS 307.44 MiB/s\n'
'[2022-10-26 14:55:55.451763] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:55:55.451810] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:55:55.451816] bdevperf '
'[2022-10-26 14:55:55.451821] '
'--no-shconf [2022-10-26 '
'14:55:55.451826] -l 0 [2022-10-26 '
'14:55:55.451831] --huge-unlink '
'[2022-10-26 14:55:55.451836] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:55:55.451841] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:55:55.451845] '
'--log-level=user1:6 [2022-10-26 '
'14:55:55.451849] --iova-mode=pa '
'[2022-10-26 14:55:55.451854] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:55:55.451859] '
'--match-allocations [2022-10-26 '
'14:55:55.451863] '
'--file-prefix=spdk_pid4572 '
'[2022-10-26 14:55:55.451868] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:55:55.573975] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:55:55.607138] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:55:55.607755] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_94.output'),
'state': {'begin': 1666781864.5490265,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 8 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.118456840515137,
'end': 1666781875.6674833,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_94.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_94.state')},
'cmd_95': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_95.output'),
'state': {'begin': 1666781875.670253,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8222548961639404,
'end': 1666781876.492508,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_95.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_95.state')},
'cmd_96': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'506563.20 IOPS 247.35 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'506563.20 IOPS 247.35 MiB/s\n'
'[2022-10-26 14:56:08.514571] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:56:08.514615] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:56:08.514621] bdevperf '
'[2022-10-26 14:56:08.514626] '
'--no-shconf [2022-10-26 '
'14:56:08.514631] -l 0 [2022-10-26 '
'14:56:08.514636] --huge-unlink '
'[2022-10-26 14:56:08.514641] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:56:08.514645] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:56:08.514649] '
'--log-level=user1:6 [2022-10-26 '
'14:56:08.514653] --iova-mode=pa '
'[2022-10-26 14:56:08.514657] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:56:08.514661] '
'--match-allocations [2022-10-26 '
'14:56:08.514667] '
'--file-prefix=spdk_pid4598 '
'[2022-10-26 14:56:08.514672] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:56:08.636693] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:56:08.679060] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:56:08.679646] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_96.output'),
'state': {'begin': 1666781877.5796776,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 8 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.161460638046265,
'end': 1666781888.7411382,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_96.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_96.state')},
'cmd_97': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_97.output'),
'state': {'begin': 1666781888.7438438,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8231239318847656,
'end': 1666781889.5669677,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_97.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_97.state')},
'cmd_98': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'482815.98 IOPS 235.75 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'482815.98 IOPS 235.75 MiB/s\n'
'[2022-10-26 14:56:21.518478] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 14:56:21.518523] [ '
'DPDK EAL parameters: [2022-10-26 '
'14:56:21.518529] bdevperf '
'[2022-10-26 14:56:21.518534] '
'--no-shconf [2022-10-26 '
'14:56:21.518538] -l 0 [2022-10-26 '
'14:56:21.518543] --huge-unlink '
'[2022-10-26 14:56:21.518547] '
'--log-level=lib.eal:6 [2022-10-26 '
'14:56:21.518551] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 14:56:21.518556] '
'--log-level=user1:6 [2022-10-26 '
'14:56:21.518560] --iova-mode=pa '
'[2022-10-26 14:56:21.518564] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 14:56:21.518568] '
'--match-allocations [2022-10-26 '
'14:56:21.518575] '
'--file-prefix=spdk_pid4623 '
'[2022-10-26 14:56:21.518580] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 14:56:21.640578] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 14:56:21.679004] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 14:56:21.679588] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_98.output'),
'state': {'begin': 1666781890.62481,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 8 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.080789804458618,
'end': 1666781901.7055998,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_98.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_98.state')},
'cmd_99': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_99.output'),
'state': {'begin': 1666781901.7083747,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.9021365642547607,
'end': 1666781902.6105113,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_99.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_99.state')}}},
'id': '001_bdevperf',
'name': 'bdevperf',
'nr': 1,
'status': {'elapsed': 783.6079878807068,
'failed': 0,
'passed': 1,
'skipped': 0,
'started': 0.0},
'uses': 'bdevperf',
'with': {'iodepths': [1, 2, 4, 8], 'iosizes': ['512'], 'repetitions': 3}}
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 150921.39 IOPS 73.69 MiB/s
=============================================================
Total : 150921.39 IOPS 73.69 MiB/s
[2022-10-26 14:45:55.003669] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:45:55.003716] [ DPDK EAL parameters: [2022-10-26 14:45:55.003722] bdevperf [2022-10-26 14:45:55.003727] --no-shconf [2022-10-26 14:45:55.003731] -l 0 [2022-10-26 14:45:55.003736] --huge-unlink [2022-10-26 14:45:55.003740] --log-level=lib.eal:6 [2022-10-26 14:45:55.003744] --log-level=lib.cryptodev:5 [2022-10-26 14:45:55.003748] --log-level=user1:6 [2022-10-26 14:45:55.003753] --iova-mode=pa [2022-10-26 14:45:55.003759] --base-virtaddr=0x200000000000 [2022-10-26 14:45:55.003763] --match-allocations [2022-10-26 14:45:55.003768] --file-prefix=spdk_pid3400 [2022-10-26 14:45:55.003772] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:45:55.125466] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:45:55.168002] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:45:55.168555] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 171436.90 IOPS 83.71 MiB/s
=============================================================
Total : 171436.90 IOPS 83.71 MiB/s
[2022-10-26 14:46:08.004494] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:46:08.004535] [ DPDK EAL parameters: [2022-10-26 14:46:08.004541] bdevperf [2022-10-26 14:46:08.004546] --no-shconf [2022-10-26 14:46:08.004550] -l 0 [2022-10-26 14:46:08.004555] --huge-unlink [2022-10-26 14:46:08.004559] --log-level=lib.eal:6 [2022-10-26 14:46:08.004564] --log-level=lib.cryptodev:5 [2022-10-26 14:46:08.004568] --log-level=user1:6 [2022-10-26 14:46:08.004573] --iova-mode=pa [2022-10-26 14:46:08.004577] --base-virtaddr=0x200000000000 [2022-10-26 14:46:08.004588] --match-allocations [2022-10-26 14:46:08.004593] --file-prefix=spdk_pid3427 [2022-10-26 14:46:08.004597] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:46:08.126140] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:46:08.166933] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:46:08.167491] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 159095.18 IOPS 77.68 MiB/s
=============================================================
Total : 159095.18 IOPS 77.68 MiB/s
[2022-10-26 14:46:21.013312] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:46:21.013360] [ DPDK EAL parameters: [2022-10-26 14:46:21.013365] bdevperf [2022-10-26 14:46:21.013370] --no-shconf [2022-10-26 14:46:21.013374] -l 0 [2022-10-26 14:46:21.013379] --huge-unlink [2022-10-26 14:46:21.013383] --log-level=lib.eal:6 [2022-10-26 14:46:21.013387] --log-level=lib.cryptodev:5 [2022-10-26 14:46:21.013391] --log-level=user1:6 [2022-10-26 14:46:21.013396] --iova-mode=pa [2022-10-26 14:46:21.013400] --base-virtaddr=0x200000000000 [2022-10-26 14:46:21.013404] --match-allocations [2022-10-26 14:46:21.013408] --file-prefix=spdk_pid3453 [2022-10-26 14:46:21.013412] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:46:21.135461] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:46:21.187017] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:46:21.187578] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 137262.72 IOPS 67.02 MiB/s
=============================================================
Total : 137262.72 IOPS 67.02 MiB/s
[2022-10-26 14:46:34.208092] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:46:34.208139] [ DPDK EAL parameters: [2022-10-26 14:46:34.208145] bdevperf [2022-10-26 14:46:34.208150] --no-shconf [2022-10-26 14:46:34.208154] -l 0 [2022-10-26 14:46:34.208158] --huge-unlink [2022-10-26 14:46:34.208164] --log-level=lib.eal:6 [2022-10-26 14:46:34.208168] --log-level=lib.cryptodev:5 [2022-10-26 14:46:34.208172] --log-level=user1:6 [2022-10-26 14:46:34.208177] --iova-mode=pa [2022-10-26 14:46:34.208181] --base-virtaddr=0x200000000000 [2022-10-26 14:46:34.208187] --match-allocations [2022-10-26 14:46:34.208191] --file-prefix=spdk_pid3478 [2022-10-26 14:46:34.208195] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:46:34.330276] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:46:34.370957] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:46:34.371536] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 138058.37 IOPS 67.41 MiB/s
=============================================================
Total : 138058.37 IOPS 67.41 MiB/s
[2022-10-26 14:46:47.231816] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:46:47.231858] [ DPDK EAL parameters: [2022-10-26 14:46:47.231864] bdevperf [2022-10-26 14:46:47.231868] --no-shconf [2022-10-26 14:46:47.231872] -l 0 [2022-10-26 14:46:47.231877] --huge-unlink [2022-10-26 14:46:47.231881] --log-level=lib.eal:6 [2022-10-26 14:46:47.231885] --log-level=lib.cryptodev:5 [2022-10-26 14:46:47.231889] --log-level=user1:6 [2022-10-26 14:46:47.231893] --iova-mode=pa [2022-10-26 14:46:47.231898] --base-virtaddr=0x200000000000 [2022-10-26 14:46:47.231902] --match-allocations [2022-10-26 14:46:47.231906] --file-prefix=spdk_pid3503 [2022-10-26 14:46:47.231910] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:46:47.353497] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:46:47.394950] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:46:47.395495] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 482862.92 IOPS 235.77 MiB/s
=============================================================
Total : 482862.92 IOPS 235.77 MiB/s
[2022-10-26 14:56:34.681870] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:56:34.681913] [ DPDK EAL parameters: [2022-10-26 14:56:34.681919] bdevperf [2022-10-26 14:56:34.681924] --no-shconf [2022-10-26 14:56:34.681928] -l 0 [2022-10-26 14:56:34.681932] --huge-unlink [2022-10-26 14:56:34.681937] --log-level=lib.eal:6 [2022-10-26 14:56:34.681941] --log-level=lib.cryptodev:5 [2022-10-26 14:56:34.681945] --log-level=user1:6 [2022-10-26 14:56:34.681949] --iova-mode=pa [2022-10-26 14:56:34.681952] --base-virtaddr=0x200000000000 [2022-10-26 14:56:34.681957] --match-allocations [2022-10-26 14:56:34.681961] --file-prefix=spdk_pid4648 [2022-10-26 14:56:34.681966] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:56:34.804031] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:56:34.842979] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:56:34.843588] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 482460.34 IOPS 235.58 MiB/s
=============================================================
Total : 482460.34 IOPS 235.58 MiB/s
[2022-10-26 14:56:47.633077] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:56:47.633125] [ DPDK EAL parameters: [2022-10-26 14:56:47.633131] bdevperf [2022-10-26 14:56:47.633136] --no-shconf [2022-10-26 14:56:47.633141] -l 0 [2022-10-26 14:56:47.633146] --huge-unlink [2022-10-26 14:56:47.633150] --log-level=lib.eal:6 [2022-10-26 14:56:47.633155] --log-level=lib.cryptodev:5 [2022-10-26 14:56:47.633160] --log-level=user1:6 [2022-10-26 14:56:47.633164] --iova-mode=pa [2022-10-26 14:56:47.633169] --base-virtaddr=0x200000000000 [2022-10-26 14:56:47.633174] --match-allocations [2022-10-26 14:56:47.633178] --file-prefix=spdk_pid4673 [2022-10-26 14:56:47.633183] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:56:47.755367] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:56:47.798960] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:56:47.799547] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 395076.98 IOPS 192.91 MiB/s
=============================================================
Total : 395076.98 IOPS 192.91 MiB/s
[2022-10-26 14:57:00.627211] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:57:00.627256] [ DPDK EAL parameters: [2022-10-26 14:57:00.627262] bdevperf [2022-10-26 14:57:00.627267] --no-shconf [2022-10-26 14:57:00.627271] -l 0 [2022-10-26 14:57:00.627276] --huge-unlink [2022-10-26 14:57:00.627280] --log-level=lib.eal:6 [2022-10-26 14:57:00.627283] --log-level=lib.cryptodev:5 [2022-10-26 14:57:00.627287] --log-level=user1:6 [2022-10-26 14:57:00.627292] --iova-mode=pa [2022-10-26 14:57:00.627297] --base-virtaddr=0x200000000000 [2022-10-26 14:57:00.627301] --match-allocations [2022-10-26 14:57:00.627305] --file-prefix=spdk_pid4698 [2022-10-26 14:57:00.627309] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:57:00.749257] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:57:00.791057] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:57:00.791650] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 392195.29 IOPS 191.50 MiB/s
=============================================================
Total : 392195.29 IOPS 191.50 MiB/s
[2022-10-26 14:57:13.666086] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:57:13.666140] [ DPDK EAL parameters: [2022-10-26 14:57:13.666148] bdevperf [2022-10-26 14:57:13.666155] --no-shconf [2022-10-26 14:57:13.666161] -l 0 [2022-10-26 14:57:13.666168] --huge-unlink [2022-10-26 14:57:13.666173] --log-level=lib.eal:6 [2022-10-26 14:57:13.666179] --log-level=lib.cryptodev:5 [2022-10-26 14:57:13.666185] --log-level=user1:6 [2022-10-26 14:57:13.666192] --iova-mode=pa [2022-10-26 14:57:13.666199] --base-virtaddr=0x200000000000 [2022-10-26 14:57:13.666206] --match-allocations [2022-10-26 14:57:13.666213] --file-prefix=spdk_pid4723 [2022-10-26 14:57:13.666219] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:57:13.788213] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:57:13.831019] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:57:13.831606] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 390867.35 IOPS 190.85 MiB/s
=============================================================
Total : 390867.35 IOPS 190.85 MiB/s
[2022-10-26 14:57:26.619274] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:57:26.619322] [ DPDK EAL parameters: [2022-10-26 14:57:26.619328] bdevperf [2022-10-26 14:57:26.619332] --no-shconf [2022-10-26 14:57:26.619337] -l 0 [2022-10-26 14:57:26.619342] --huge-unlink [2022-10-26 14:57:26.619348] --log-level=lib.eal:6 [2022-10-26 14:57:26.619353] --log-level=lib.cryptodev:5 [2022-10-26 14:57:26.619358] --log-level=user1:6 [2022-10-26 14:57:26.619362] --iova-mode=pa [2022-10-26 14:57:26.619366] --base-virtaddr=0x200000000000 [2022-10-26 14:57:26.619371] --match-allocations [2022-10-26 14:57:26.619376] --file-prefix=spdk_pid4748 [2022-10-26 14:57:26.619380] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:57:26.741441] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:57:26.791036] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:57:26.791643] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 404566.90 IOPS 197.54 MiB/s
=============================================================
Total : 404566.90 IOPS 197.54 MiB/s
[2022-10-26 14:57:39.566766] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:57:39.566809] [ DPDK EAL parameters: [2022-10-26 14:57:39.566815] bdevperf [2022-10-26 14:57:39.566821] --no-shconf [2022-10-26 14:57:39.566825] -l 0 [2022-10-26 14:57:39.566829] --huge-unlink [2022-10-26 14:57:39.566834] --log-level=lib.eal:6 [2022-10-26 14:57:39.566838] --log-level=lib.cryptodev:5 [2022-10-26 14:57:39.566842] --log-level=user1:6 [2022-10-26 14:57:39.566846] --iova-mode=pa [2022-10-26 14:57:39.566850] --base-virtaddr=0x200000000000 [2022-10-26 14:57:39.566855] --match-allocations [2022-10-26 14:57:39.566859] --file-prefix=spdk_pid4773 [2022-10-26 14:57:39.566863] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:57:39.688845] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:57:39.731022] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:57:39.731598] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 404402.85 IOPS 197.46 MiB/s
=============================================================
Total : 404402.85 IOPS 197.46 MiB/s
[2022-10-26 14:57:52.662483] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:57:52.662525] [ DPDK EAL parameters: [2022-10-26 14:57:52.662531] bdevperf [2022-10-26 14:57:52.662536] --no-shconf [2022-10-26 14:57:52.662540] -l 0 [2022-10-26 14:57:52.662545] --huge-unlink [2022-10-26 14:57:52.662549] --log-level=lib.eal:6 [2022-10-26 14:57:52.662553] --log-level=lib.cryptodev:5 [2022-10-26 14:57:52.662557] --log-level=user1:6 [2022-10-26 14:57:52.662561] --iova-mode=pa [2022-10-26 14:57:52.662565] --base-virtaddr=0x200000000000 [2022-10-26 14:57:52.662570] --match-allocations [2022-10-26 14:57:52.662574] --file-prefix=spdk_pid4799 [2022-10-26 14:57:52.662579] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:57:52.784566] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:57:52.827141] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:57:52.827734] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 403868.55 IOPS 197.20 MiB/s
=============================================================
Total : 403868.55 IOPS 197.20 MiB/s
[2022-10-26 14:58:05.759802] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:58:05.759849] [ DPDK EAL parameters: [2022-10-26 14:58:05.759855] bdevperf [2022-10-26 14:58:05.759860] --no-shconf [2022-10-26 14:58:05.759864] -l 0 [2022-10-26 14:58:05.759869] --huge-unlink [2022-10-26 14:58:05.759874] --log-level=lib.eal:6 [2022-10-26 14:58:05.759878] --log-level=lib.cryptodev:5 [2022-10-26 14:58:05.759883] --log-level=user1:6 [2022-10-26 14:58:05.759887] --iova-mode=pa [2022-10-26 14:58:05.759891] --base-virtaddr=0x200000000000 [2022-10-26 14:58:05.759895] --match-allocations [2022-10-26 14:58:05.759900] --file-prefix=spdk_pid4825 [2022-10-26 14:58:05.759904] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:58:05.881869] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:58:05.919038] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:58:05.919619] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 382108.42 IOPS 186.58 MiB/s
=============================================================
Total : 382108.42 IOPS 186.58 MiB/s
[2022-10-26 14:58:18.828990] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:58:18.829036] [ DPDK EAL parameters: [2022-10-26 14:58:18.829042] bdevperf [2022-10-26 14:58:18.829047] --no-shconf [2022-10-26 14:58:18.829051] -l 0 [2022-10-26 14:58:18.829056] --huge-unlink [2022-10-26 14:58:18.829061] --log-level=lib.eal:6 [2022-10-26 14:58:18.829065] --log-level=lib.cryptodev:5 [2022-10-26 14:58:18.829070] --log-level=user1:6 [2022-10-26 14:58:18.829074] --iova-mode=pa [2022-10-26 14:58:18.829078] --base-virtaddr=0x200000000000 [2022-10-26 14:58:18.829083] --match-allocations [2022-10-26 14:58:18.829088] --file-prefix=spdk_pid4850 [2022-10-26 14:58:18.829092] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:58:18.951199] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:58:18.995101] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:58:18.995684] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 380577.08 IOPS 185.83 MiB/s
=============================================================
Total : 380577.08 IOPS 185.83 MiB/s
[2022-10-26 14:58:32.287850] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:58:32.287904] [ DPDK EAL parameters: [2022-10-26 14:58:32.287910] bdevperf [2022-10-26 14:58:32.287915] --no-shconf [2022-10-26 14:58:32.287920] -l 0 [2022-10-26 14:58:32.287924] --huge-unlink [2022-10-26 14:58:32.287929] --log-level=lib.eal:6 [2022-10-26 14:58:32.287933] --log-level=lib.cryptodev:5 [2022-10-26 14:58:32.287937] --log-level=user1:6 [2022-10-26 14:58:32.287942] --iova-mode=pa [2022-10-26 14:58:32.287946] --base-virtaddr=0x200000000000 [2022-10-26 14:58:32.287950] --match-allocations [2022-10-26 14:58:32.287955] --file-prefix=spdk_pid4875 [2022-10-26 14:58:32.287960] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:58:32.409866] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:58:32.450980] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:58:32.451586] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 137312.00 IOPS 67.05 MiB/s
=============================================================
Total : 137312.00 IOPS 67.05 MiB/s
[2022-10-26 14:47:00.239018] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:47:00.239062] [ DPDK EAL parameters: [2022-10-26 14:47:00.239067] bdevperf [2022-10-26 14:47:00.239073] --no-shconf [2022-10-26 14:47:00.239077] -l 0 [2022-10-26 14:47:00.239082] --huge-unlink [2022-10-26 14:47:00.239086] --log-level=lib.eal:6 [2022-10-26 14:47:00.239091] --log-level=lib.cryptodev:5 [2022-10-26 14:47:00.239096] --log-level=user1:6 [2022-10-26 14:47:00.239100] --iova-mode=pa [2022-10-26 14:47:00.239104] --base-virtaddr=0x200000000000 [2022-10-26 14:47:00.239108] --match-allocations [2022-10-26 14:47:00.239112] --file-prefix=spdk_pid3528 [2022-10-26 14:47:00.239117] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:47:00.360863] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:47:00.406981] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:47:00.407550] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 379284.08 IOPS 185.20 MiB/s
=============================================================
Total : 379284.08 IOPS 185.20 MiB/s
[2022-10-26 14:58:45.398827] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:58:45.398870] [ DPDK EAL parameters: [2022-10-26 14:58:45.398876] bdevperf [2022-10-26 14:58:45.398881] --no-shconf [2022-10-26 14:58:45.398885] -l 0 [2022-10-26 14:58:45.398890] --huge-unlink [2022-10-26 14:58:45.398895] --log-level=lib.eal:6 [2022-10-26 14:58:45.398899] --log-level=lib.cryptodev:5 [2022-10-26 14:58:45.398904] --log-level=user1:6 [2022-10-26 14:58:45.398908] --iova-mode=pa [2022-10-26 14:58:45.398912] --base-virtaddr=0x200000000000 [2022-10-26 14:58:45.398918] --match-allocations [2022-10-26 14:58:45.398924] --file-prefix=spdk_pid4900 [2022-10-26 14:58:45.398929] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:58:45.521201] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:58:45.571063] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:58:45.571640] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 156589.41 IOPS 76.46 MiB/s
=============================================================
Total : 156589.41 IOPS 76.46 MiB/s
[2022-10-26 14:47:13.264251] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:47:13.264293] [ DPDK EAL parameters: [2022-10-26 14:47:13.264298] bdevperf [2022-10-26 14:47:13.264303] --no-shconf [2022-10-26 14:47:13.264307] -l 0 [2022-10-26 14:47:13.264311] --huge-unlink [2022-10-26 14:47:13.264316] --log-level=lib.eal:6 [2022-10-26 14:47:13.264320] --log-level=lib.cryptodev:5 [2022-10-26 14:47:13.264324] --log-level=user1:6 [2022-10-26 14:47:13.264328] --iova-mode=pa [2022-10-26 14:47:13.264332] --base-virtaddr=0x200000000000 [2022-10-26 14:47:13.264336] --match-allocations [2022-10-26 14:47:13.264340] --file-prefix=spdk_pid3553 [2022-10-26 14:47:13.264344] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:47:13.386330] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:47:13.430961] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:47:13.431535] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 156437.11 IOPS 76.39 MiB/s
=============================================================
Total : 156437.11 IOPS 76.39 MiB/s
[2022-10-26 14:47:26.307921] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:47:26.307971] [ DPDK EAL parameters: [2022-10-26 14:47:26.307978] bdevperf [2022-10-26 14:47:26.307985] --no-shconf [2022-10-26 14:47:26.307992] -l 0 [2022-10-26 14:47:26.307997] --huge-unlink [2022-10-26 14:47:26.308003] --log-level=lib.eal:6 [2022-10-26 14:47:26.308009] --log-level=lib.cryptodev:5 [2022-10-26 14:47:26.308016] --log-level=user1:6 [2022-10-26 14:47:26.308022] --iova-mode=pa [2022-10-26 14:47:26.308028] --base-virtaddr=0x200000000000 [2022-10-26 14:47:26.308034] --match-allocations [2022-10-26 14:47:26.308041] --file-prefix=spdk_pid3578 [2022-10-26 14:47:26.308053] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:47:26.429664] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:47:26.462979] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:47:26.463535] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 156357.41 IOPS 76.35 MiB/s
=============================================================
Total : 156357.41 IOPS 76.35 MiB/s
[2022-10-26 14:47:39.245614] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:47:39.245656] [ DPDK EAL parameters: [2022-10-26 14:47:39.245662] bdevperf [2022-10-26 14:47:39.245667] --no-shconf [2022-10-26 14:47:39.245672] -l 0 [2022-10-26 14:47:39.245676] --huge-unlink [2022-10-26 14:47:39.245681] --log-level=lib.eal:6 [2022-10-26 14:47:39.245685] --log-level=lib.cryptodev:5 [2022-10-26 14:47:39.245689] --log-level=user1:6 [2022-10-26 14:47:39.245694] --iova-mode=pa [2022-10-26 14:47:39.245698] --base-virtaddr=0x200000000000 [2022-10-26 14:47:39.245703] --match-allocations [2022-10-26 14:47:39.245708] --file-prefix=spdk_pid3603 [2022-10-26 14:47:39.245712] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:47:39.367532] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:47:39.411007] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:47:39.411561] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
rm: cannot remove '/tmp/spdk_bdev_aio_foo.conf': No such file or directory
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 154952.02 IOPS 75.66 MiB/s
=============================================================
Total : 154952.02 IOPS 75.66 MiB/s
[2022-10-26 14:47:52.225176] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:47:52.225221] [ DPDK EAL parameters: [2022-10-26 14:47:52.225227] bdevperf [2022-10-26 14:47:52.225232] --no-shconf [2022-10-26 14:47:52.225236] -l 0 [2022-10-26 14:47:52.225240] --huge-unlink [2022-10-26 14:47:52.225245] --log-level=lib.eal:6 [2022-10-26 14:47:52.225249] --log-level=lib.cryptodev:5 [2022-10-26 14:47:52.225253] --log-level=user1:6 [2022-10-26 14:47:52.225257] --iova-mode=pa [2022-10-26 14:47:52.225261] --base-virtaddr=0x200000000000 [2022-10-26 14:47:52.225265] --match-allocations [2022-10-26 14:47:52.225270] --file-prefix=spdk_pid3628 [2022-10-26 14:47:52.225276] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:47:52.346983] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:47:52.398946] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:47:52.399501] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 154636.84 IOPS 75.51 MiB/s
=============================================================
Total : 154636.84 IOPS 75.51 MiB/s
[2022-10-26 14:48:05.239169] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:48:05.239212] [ DPDK EAL parameters: [2022-10-26 14:48:05.239218] bdevperf [2022-10-26 14:48:05.239223] --no-shconf [2022-10-26 14:48:05.239228] -l 0 [2022-10-26 14:48:05.239233] --huge-unlink [2022-10-26 14:48:05.239237] --log-level=lib.eal:6 [2022-10-26 14:48:05.239241] --log-level=lib.cryptodev:5 [2022-10-26 14:48:05.239246] --log-level=user1:6 [2022-10-26 14:48:05.239250] --iova-mode=pa [2022-10-26 14:48:05.239254] --base-virtaddr=0x200000000000 [2022-10-26 14:48:05.239258] --match-allocations [2022-10-26 14:48:05.239264] --file-prefix=spdk_pid3653 [2022-10-26 14:48:05.239268] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:48:05.361024] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:48:05.410953] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:48:05.411512] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 154684.51 IOPS 75.53 MiB/s
=============================================================
Total : 154684.51 IOPS 75.53 MiB/s
[2022-10-26 14:48:18.287627] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:48:18.287671] [ DPDK EAL parameters: [2022-10-26 14:48:18.287677] bdevperf [2022-10-26 14:48:18.287682] --no-shconf [2022-10-26 14:48:18.287686] -l 0 [2022-10-26 14:48:18.287691] --huge-unlink [2022-10-26 14:48:18.287695] --log-level=lib.eal:6 [2022-10-26 14:48:18.287700] --log-level=lib.cryptodev:5 [2022-10-26 14:48:18.287703] --log-level=user1:6 [2022-10-26 14:48:18.287708] --iova-mode=pa [2022-10-26 14:48:18.287712] --base-virtaddr=0x200000000000 [2022-10-26 14:48:18.287716] --match-allocations [2022-10-26 14:48:18.287721] --file-prefix=spdk_pid3679 [2022-10-26 14:48:18.287727] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:48:18.409256] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:48:18.452516] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:48:18.453072] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
rm: cannot remove '/tmp/spdk_bdev_xnvme_libaio.conf': No such file or directory
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 145398.43 IOPS 71.00 MiB/s
=============================================================
Total : 145398.43 IOPS 71.00 MiB/s
[2022-10-26 14:48:31.500046] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:48:31.500091] [ DPDK EAL parameters: [2022-10-26 14:48:31.500096] bdevperf [2022-10-26 14:48:31.500102] --no-shconf [2022-10-26 14:48:31.500106] -l 0 [2022-10-26 14:48:31.500111] --huge-unlink [2022-10-26 14:48:31.500115] --log-level=lib.eal:6 [2022-10-26 14:48:31.500119] --log-level=lib.cryptodev:5 [2022-10-26 14:48:31.500123] --log-level=user1:6 [2022-10-26 14:48:31.500128] --iova-mode=pa [2022-10-26 14:48:31.500132] --base-virtaddr=0x200000000000 [2022-10-26 14:48:31.500136] --match-allocations [2022-10-26 14:48:31.500140] --file-prefix=spdk_pid3704 [2022-10-26 14:48:31.500144] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:48:31.621779] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:48:31.662966] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:48:31.663515] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 146366.04 IOPS 71.47 MiB/s
=============================================================
Total : 146366.04 IOPS 71.47 MiB/s
[2022-10-26 14:48:44.543718] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:48:44.543768] [ DPDK EAL parameters: [2022-10-26 14:48:44.543775] bdevperf [2022-10-26 14:48:44.543780] --no-shconf [2022-10-26 14:48:44.543787] -l 0 [2022-10-26 14:48:44.543793] --huge-unlink [2022-10-26 14:48:44.543799] --log-level=lib.eal:6 [2022-10-26 14:48:44.543805] --log-level=lib.cryptodev:5 [2022-10-26 14:48:44.543810] --log-level=user1:6 [2022-10-26 14:48:44.543816] --iova-mode=pa [2022-10-26 14:48:44.543822] --base-virtaddr=0x200000000000 [2022-10-26 14:48:44.543828] --match-allocations [2022-10-26 14:48:44.543834] --file-prefix=spdk_pid3729 [2022-10-26 14:48:44.543839] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:48:44.665842] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:48:44.707021] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:48:44.707577] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 146618.86 IOPS 71.59 MiB/s
=============================================================
Total : 146618.86 IOPS 71.59 MiB/s
[2022-10-26 14:48:57.629719] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:48:57.629762] [ DPDK EAL parameters: [2022-10-26 14:48:57.629768] bdevperf [2022-10-26 14:48:57.629773] --no-shconf [2022-10-26 14:48:57.629778] -l 0 [2022-10-26 14:48:57.629782] --huge-unlink [2022-10-26 14:48:57.629785] --log-level=lib.eal:6 [2022-10-26 14:48:57.629790] --log-level=lib.cryptodev:5 [2022-10-26 14:48:57.629793] --log-level=user1:6 [2022-10-26 14:48:57.629798] --iova-mode=pa [2022-10-26 14:48:57.629801] --base-virtaddr=0x200000000000 [2022-10-26 14:48:57.629805] --match-allocations [2022-10-26 14:48:57.629809] --file-prefix=spdk_pid3755 [2022-10-26 14:48:57.629813] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:48:57.752067] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:48:57.794987] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:48:57.795541] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 261816.82 IOPS 127.84 MiB/s
=============================================================
Total : 261816.82 IOPS 127.84 MiB/s
[2022-10-26 14:49:10.728753] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:49:10.728804] [ DPDK EAL parameters: [2022-10-26 14:49:10.728810] bdevperf [2022-10-26 14:49:10.728814] --no-shconf [2022-10-26 14:49:10.728818] -l 0 [2022-10-26 14:49:10.728823] --huge-unlink [2022-10-26 14:49:10.728827] --log-level=lib.eal:6 [2022-10-26 14:49:10.728831] --log-level=lib.cryptodev:5 [2022-10-26 14:49:10.728835] --log-level=user1:6 [2022-10-26 14:49:10.728839] --iova-mode=pa [2022-10-26 14:49:10.728843] --base-virtaddr=0x200000000000 [2022-10-26 14:49:10.728847] --match-allocations [2022-10-26 14:49:10.728851] --file-prefix=spdk_pid3780 [2022-10-26 14:49:10.728857] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:49:10.850519] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:49:10.894989] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:49:10.895537] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 261099.59 IOPS 127.49 MiB/s
=============================================================
Total : 261099.59 IOPS 127.49 MiB/s
[2022-10-26 14:49:23.852529] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:49:23.852573] [ DPDK EAL parameters: [2022-10-26 14:49:23.852579] bdevperf [2022-10-26 14:49:23.852583] --no-shconf [2022-10-26 14:49:23.852589] -l 0 [2022-10-26 14:49:23.852594] --huge-unlink [2022-10-26 14:49:23.852599] --log-level=lib.eal:6 [2022-10-26 14:49:23.852603] --log-level=lib.cryptodev:5 [2022-10-26 14:49:23.852608] --log-level=user1:6 [2022-10-26 14:49:23.852612] --iova-mode=pa [2022-10-26 14:49:23.852616] --base-virtaddr=0x200000000000 [2022-10-26 14:49:23.852621] --match-allocations [2022-10-26 14:49:23.852626] --file-prefix=spdk_pid3805 [2022-10-26 14:49:23.852632] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:49:23.974359] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:49:24.027041] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:49:24.027602] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 260889.10 IOPS 127.39 MiB/s
=============================================================
Total : 260889.10 IOPS 127.39 MiB/s
[2022-10-26 14:49:36.826143] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:49:36.826185] [ DPDK EAL parameters: [2022-10-26 14:49:36.826191] bdevperf [2022-10-26 14:49:36.826195] --no-shconf [2022-10-26 14:49:36.826200] -l 0 [2022-10-26 14:49:36.826204] --huge-unlink [2022-10-26 14:49:36.826209] --log-level=lib.eal:6 [2022-10-26 14:49:36.826213] --log-level=lib.cryptodev:5 [2022-10-26 14:49:36.826218] --log-level=user1:6 [2022-10-26 14:49:36.826222] --iova-mode=pa [2022-10-26 14:49:36.826226] --base-virtaddr=0x200000000000 [2022-10-26 14:49:36.826230] --match-allocations [2022-10-26 14:49:36.826235] --file-prefix=spdk_pid3830 [2022-10-26 14:49:36.826239] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:49:36.947892] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:49:36.991044] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:49:36.991602] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 230826.32 IOPS 112.71 MiB/s
=============================================================
Total : 230826.32 IOPS 112.71 MiB/s
[2022-10-26 14:49:49.866229] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:49:49.866273] [ DPDK EAL parameters: [2022-10-26 14:49:49.866279] bdevperf [2022-10-26 14:49:49.866283] --no-shconf [2022-10-26 14:49:49.866288] -l 0 [2022-10-26 14:49:49.866293] --huge-unlink [2022-10-26 14:49:49.866298] --log-level=lib.eal:6 [2022-10-26 14:49:49.866303] --log-level=lib.cryptodev:5 [2022-10-26 14:49:49.866307] --log-level=user1:6 [2022-10-26 14:49:49.866311] --iova-mode=pa [2022-10-26 14:49:49.866316] --base-virtaddr=0x200000000000 [2022-10-26 14:49:49.866321] --match-allocations [2022-10-26 14:49:49.866325] --file-prefix=spdk_pid3855 [2022-10-26 14:49:49.866330] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:49:49.987927] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:49:50.031010] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:49:50.031564] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 229739.44 IOPS 112.18 MiB/s
=============================================================
Total : 229739.44 IOPS 112.18 MiB/s
[2022-10-26 14:50:02.780961] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:50:02.781004] [ DPDK EAL parameters: [2022-10-26 14:50:02.781010] bdevperf [2022-10-26 14:50:02.781015] --no-shconf [2022-10-26 14:50:02.781020] -l 0 [2022-10-26 14:50:02.781026] --huge-unlink [2022-10-26 14:50:02.781030] --log-level=lib.eal:6 [2022-10-26 14:50:02.781034] --log-level=lib.cryptodev:5 [2022-10-26 14:50:02.781038] --log-level=user1:6 [2022-10-26 14:50:02.781042] --iova-mode=pa [2022-10-26 14:50:02.781046] --base-virtaddr=0x200000000000 [2022-10-26 14:50:02.781050] --match-allocations [2022-10-26 14:50:02.781060] --file-prefix=spdk_pid3880 [2022-10-26 14:50:02.781064] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:50:02.902700] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:50:02.946942] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:50:02.947497] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 231226.96 IOPS 112.90 MiB/s
=============================================================
Total : 231226.96 IOPS 112.90 MiB/s
[2022-10-26 14:50:15.813398] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:50:15.813439] [ DPDK EAL parameters: [2022-10-26 14:50:15.813445] bdevperf [2022-10-26 14:50:15.813450] --no-shconf [2022-10-26 14:50:15.813455] -l 0 [2022-10-26 14:50:15.813460] --huge-unlink [2022-10-26 14:50:15.813464] --log-level=lib.eal:6 [2022-10-26 14:50:15.813470] --log-level=lib.cryptodev:5 [2022-10-26 14:50:15.813474] --log-level=user1:6 [2022-10-26 14:50:15.813478] --iova-mode=pa [2022-10-26 14:50:15.813482] --base-virtaddr=0x200000000000 [2022-10-26 14:50:15.813487] --match-allocations [2022-10-26 14:50:15.813491] --file-prefix=spdk_pid3906 [2022-10-26 14:50:15.813495] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:50:15.935094] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:50:15.978913] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:50:15.979453] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 282909.29 IOPS 138.14 MiB/s
=============================================================
Total : 282909.29 IOPS 138.14 MiB/s
[2022-10-26 14:50:28.780387] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:50:28.780434] [ DPDK EAL parameters: [2022-10-26 14:50:28.780440] bdevperf [2022-10-26 14:50:28.780445] --no-shconf [2022-10-26 14:50:28.780449] -l 0 [2022-10-26 14:50:28.780453] --huge-unlink [2022-10-26 14:50:28.780458] --log-level=lib.eal:6 [2022-10-26 14:50:28.780463] --log-level=lib.cryptodev:5 [2022-10-26 14:50:28.780467] --log-level=user1:6 [2022-10-26 14:50:28.780471] --iova-mode=pa [2022-10-26 14:50:28.780475] --base-virtaddr=0x200000000000 [2022-10-26 14:50:28.780479] --match-allocations [2022-10-26 14:50:28.780485] --file-prefix=spdk_pid3931 [2022-10-26 14:50:28.780491] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:50:28.902341] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:50:28.948483] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:50:28.949036] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 283679.60 IOPS 138.52 MiB/s
=============================================================
Total : 283679.60 IOPS 138.52 MiB/s
[2022-10-26 14:50:41.753061] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:50:41.753108] [ DPDK EAL parameters: [2022-10-26 14:50:41.753113] bdevperf [2022-10-26 14:50:41.753119] --no-shconf [2022-10-26 14:50:41.753123] -l 0 [2022-10-26 14:50:41.753128] --huge-unlink [2022-10-26 14:50:41.753132] --log-level=lib.eal:6 [2022-10-26 14:50:41.753138] --log-level=lib.cryptodev:5 [2022-10-26 14:50:41.753142] --log-level=user1:6 [2022-10-26 14:50:41.753146] --iova-mode=pa [2022-10-26 14:50:41.753150] --base-virtaddr=0x200000000000 [2022-10-26 14:50:41.753154] --match-allocations [2022-10-26 14:50:41.753160] --file-prefix=spdk_pid3957 [2022-10-26 14:50:41.753164] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:50:41.874953] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:50:41.915040] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:50:41.915590] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 283155.55 IOPS 138.26 MiB/s
=============================================================
Total : 283155.55 IOPS 138.26 MiB/s
[2022-10-26 14:50:54.724079] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:50:54.724121] [ DPDK EAL parameters: [2022-10-26 14:50:54.724127] bdevperf [2022-10-26 14:50:54.724132] --no-shconf [2022-10-26 14:50:54.724137] -l 0 [2022-10-26 14:50:54.724141] --huge-unlink [2022-10-26 14:50:54.724146] --log-level=lib.eal:6 [2022-10-26 14:50:54.724149] --log-level=lib.cryptodev:5 [2022-10-26 14:50:54.724154] --log-level=user1:6 [2022-10-26 14:50:54.724157] --iova-mode=pa [2022-10-26 14:50:54.724161] --base-virtaddr=0x200000000000 [2022-10-26 14:50:54.724165] --match-allocations [2022-10-26 14:50:54.724169] --file-prefix=spdk_pid3984 [2022-10-26 14:50:54.724175] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:50:54.845795] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:50:54.887066] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:50:54.887623] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 281466.49 IOPS 137.43 MiB/s
=============================================================
Total : 281466.49 IOPS 137.43 MiB/s
[2022-10-26 14:51:07.680050] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:51:07.680094] [ DPDK EAL parameters: [2022-10-26 14:51:07.680101] bdevperf [2022-10-26 14:51:07.680106] --no-shconf [2022-10-26 14:51:07.680111] -l 0 [2022-10-26 14:51:07.680118] --huge-unlink [2022-10-26 14:51:07.680126] --log-level=lib.eal:6 [2022-10-26 14:51:07.680132] --log-level=lib.cryptodev:5 [2022-10-26 14:51:07.680137] --log-level=user1:6 [2022-10-26 14:51:07.680142] --iova-mode=pa [2022-10-26 14:51:07.680149] --base-virtaddr=0x200000000000 [2022-10-26 14:51:07.680154] --match-allocations [2022-10-26 14:51:07.680160] --file-prefix=spdk_pid4010 [2022-10-26 14:51:07.680167] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:51:07.801780] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:51:07.839036] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:51:07.839594] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 280747.16 IOPS 137.08 MiB/s
=============================================================
Total : 280747.16 IOPS 137.08 MiB/s
[2022-10-26 14:51:20.723914] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:51:20.723958] [ DPDK EAL parameters: [2022-10-26 14:51:20.723964] bdevperf [2022-10-26 14:51:20.723968] --no-shconf [2022-10-26 14:51:20.723973] -l 0 [2022-10-26 14:51:20.723979] --huge-unlink [2022-10-26 14:51:20.723983] --log-level=lib.eal:6 [2022-10-26 14:51:20.723988] --log-level=lib.cryptodev:5 [2022-10-26 14:51:20.723992] --log-level=user1:6 [2022-10-26 14:51:20.723997] --iova-mode=pa [2022-10-26 14:51:20.724001] --base-virtaddr=0x200000000000 [2022-10-26 14:51:20.724004] --match-allocations [2022-10-26 14:51:20.724009] --file-prefix=spdk_pid4036 [2022-10-26 14:51:20.724014] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:51:20.845500] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:51:20.895019] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:51:20.895578] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 280777.96 IOPS 137.10 MiB/s
=============================================================
Total : 280777.96 IOPS 137.10 MiB/s
[2022-10-26 14:51:33.686533] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:51:33.686578] [ DPDK EAL parameters: [2022-10-26 14:51:33.686584] bdevperf [2022-10-26 14:51:33.686589] --no-shconf [2022-10-26 14:51:33.686594] -l 0 [2022-10-26 14:51:33.686598] --huge-unlink [2022-10-26 14:51:33.686603] --log-level=lib.eal:6 [2022-10-26 14:51:33.686608] --log-level=lib.cryptodev:5 [2022-10-26 14:51:33.686612] --log-level=user1:6 [2022-10-26 14:51:33.686616] --iova-mode=pa [2022-10-26 14:51:33.686620] --base-virtaddr=0x200000000000 [2022-10-26 14:51:33.686625] --match-allocations [2022-10-26 14:51:33.686630] --file-prefix=spdk_pid4063 [2022-10-26 14:51:33.686635] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:51:33.808399] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:51:33.851003] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:51:33.851560] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 267085.41 IOPS 130.41 MiB/s
=============================================================
Total : 267085.41 IOPS 130.41 MiB/s
[2022-10-26 14:51:46.743969] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:51:46.744011] [ DPDK EAL parameters: [2022-10-26 14:51:46.744017] bdevperf [2022-10-26 14:51:46.744021] --no-shconf [2022-10-26 14:51:46.744026] -l 0 [2022-10-26 14:51:46.744030] --huge-unlink [2022-10-26 14:51:46.744034] --log-level=lib.eal:6 [2022-10-26 14:51:46.744038] --log-level=lib.cryptodev:5 [2022-10-26 14:51:46.744044] --log-level=user1:6 [2022-10-26 14:51:46.744048] --iova-mode=pa [2022-10-26 14:51:46.744053] --base-virtaddr=0x200000000000 [2022-10-26 14:51:46.744058] --match-allocations [2022-10-26 14:51:46.744064] --file-prefix=spdk_pid4090 [2022-10-26 14:51:46.744068] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:51:46.866027] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:51:46.903064] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:51:46.903629] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 267177.11 IOPS 130.46 MiB/s
=============================================================
Total : 267177.11 IOPS 130.46 MiB/s
[2022-10-26 14:51:59.778212] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:51:59.778256] [ DPDK EAL parameters: [2022-10-26 14:51:59.778262] bdevperf [2022-10-26 14:51:59.778267] --no-shconf [2022-10-26 14:51:59.778271] -l 0 [2022-10-26 14:51:59.778276] --huge-unlink [2022-10-26 14:51:59.778281] --log-level=lib.eal:6 [2022-10-26 14:51:59.778285] --log-level=lib.cryptodev:5 [2022-10-26 14:51:59.778291] --log-level=user1:6 [2022-10-26 14:51:59.778295] --iova-mode=pa [2022-10-26 14:51:59.778299] --base-virtaddr=0x200000000000 [2022-10-26 14:51:59.778303] --match-allocations [2022-10-26 14:51:59.778307] --file-prefix=spdk_pid4115 [2022-10-26 14:51:59.778313] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:51:59.900002] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:51:59.951001] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:51:59.951562] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 266611.68 IOPS 130.18 MiB/s
=============================================================
Total : 266611.68 IOPS 130.18 MiB/s
[2022-10-26 14:52:12.820906] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:52:12.820957] [ DPDK EAL parameters: [2022-10-26 14:52:12.820965] bdevperf [2022-10-26 14:52:12.820972] --no-shconf [2022-10-26 14:52:12.820978] -l 0 [2022-10-26 14:52:12.820984] --huge-unlink [2022-10-26 14:52:12.820990] --log-level=lib.eal:6 [2022-10-26 14:52:12.820997] --log-level=lib.cryptodev:5 [2022-10-26 14:52:12.821003] --log-level=user1:6 [2022-10-26 14:52:12.821011] --iova-mode=pa [2022-10-26 14:52:12.821019] --base-virtaddr=0x200000000000 [2022-10-26 14:52:12.821025] --match-allocations [2022-10-26 14:52:12.821032] --file-prefix=spdk_pid4140 [2022-10-26 14:52:12.821039] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:52:12.942842] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:52:12.987011] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:52:12.987571] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 409829.09 IOPS 200.11 MiB/s
=============================================================
Total : 409829.09 IOPS 200.11 MiB/s
[2022-10-26 14:52:25.997001] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:52:25.997050] [ DPDK EAL parameters: [2022-10-26 14:52:25.997056] bdevperf [2022-10-26 14:52:25.997060] --no-shconf [2022-10-26 14:52:25.997064] -l 0 [2022-10-26 14:52:25.997069] --huge-unlink [2022-10-26 14:52:25.997074] --log-level=lib.eal:6 [2022-10-26 14:52:25.997078] --log-level=lib.cryptodev:5 [2022-10-26 14:52:25.997083] --log-level=user1:6 [2022-10-26 14:52:25.997087] --iova-mode=pa [2022-10-26 14:52:25.997091] --base-virtaddr=0x200000000000 [2022-10-26 14:52:25.997095] --match-allocations [2022-10-26 14:52:25.997100] --file-prefix=spdk_pid4165 [2022-10-26 14:52:25.997105] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:52:26.118822] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:52:26.162992] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:52:26.163553] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 453036.84 IOPS 221.21 MiB/s
=============================================================
Total : 453036.84 IOPS 221.21 MiB/s
[2022-10-26 14:52:39.039669] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:52:39.039712] [ DPDK EAL parameters: [2022-10-26 14:52:39.039717] bdevperf [2022-10-26 14:52:39.039722] --no-shconf [2022-10-26 14:52:39.039726] -l 0 [2022-10-26 14:52:39.039731] --huge-unlink [2022-10-26 14:52:39.039736] --log-level=lib.eal:6 [2022-10-26 14:52:39.039740] --log-level=lib.cryptodev:5 [2022-10-26 14:52:39.039745] --log-level=user1:6 [2022-10-26 14:52:39.039749] --iova-mode=pa [2022-10-26 14:52:39.039752] --base-virtaddr=0x200000000000 [2022-10-26 14:52:39.039757] --match-allocations [2022-10-26 14:52:39.039761] --file-prefix=spdk_pid4190 [2022-10-26 14:52:39.039766] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:52:39.161430] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:52:39.203055] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:52:39.203611] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 493618.05 IOPS 241.02 MiB/s
=============================================================
Total : 493618.05 IOPS 241.02 MiB/s
[2022-10-26 14:52:52.548321] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:52:52.548367] [ DPDK EAL parameters: [2022-10-26 14:52:52.548372] bdevperf [2022-10-26 14:52:52.548377] --no-shconf [2022-10-26 14:52:52.548381] -l 0 [2022-10-26 14:52:52.548385] --huge-unlink [2022-10-26 14:52:52.548390] --log-level=lib.eal:6 [2022-10-26 14:52:52.548394] --log-level=lib.cryptodev:5 [2022-10-26 14:52:52.548398] --log-level=user1:6 [2022-10-26 14:52:52.548402] --iova-mode=pa [2022-10-26 14:52:52.548406] --base-virtaddr=0x200000000000 [2022-10-26 14:52:52.548410] --match-allocations [2022-10-26 14:52:52.548415] --file-prefix=spdk_pid4215 [2022-10-26 14:52:52.548422] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:52:52.669991] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:52:52.711005] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:52:52.711571] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 284288.83 IOPS 138.81 MiB/s
=============================================================
Total : 284288.83 IOPS 138.81 MiB/s
[2022-10-26 14:53:05.541424] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:53:05.541465] [ DPDK EAL parameters: [2022-10-26 14:53:05.541471] bdevperf [2022-10-26 14:53:05.541475] --no-shconf [2022-10-26 14:53:05.541480] -l 0 [2022-10-26 14:53:05.541484] --huge-unlink [2022-10-26 14:53:05.541488] --log-level=lib.eal:6 [2022-10-26 14:53:05.541493] --log-level=lib.cryptodev:5 [2022-10-26 14:53:05.541497] --log-level=user1:6 [2022-10-26 14:53:05.541501] --iova-mode=pa [2022-10-26 14:53:05.541506] --base-virtaddr=0x200000000000 [2022-10-26 14:53:05.541511] --match-allocations [2022-10-26 14:53:05.541515] --file-prefix=spdk_pid4240 [2022-10-26 14:53:05.541520] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:53:05.663378] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:53:05.706987] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:53:05.707569] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 283826.39 IOPS 138.59 MiB/s
=============================================================
Total : 283826.39 IOPS 138.59 MiB/s
[2022-10-26 14:53:18.550947] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:53:18.550994] [ DPDK EAL parameters: [2022-10-26 14:53:18.551000] bdevperf [2022-10-26 14:53:18.551004] --no-shconf [2022-10-26 14:53:18.551009] -l 0 [2022-10-26 14:53:18.551013] --huge-unlink [2022-10-26 14:53:18.551018] --log-level=lib.eal:6 [2022-10-26 14:53:18.551022] --log-level=lib.cryptodev:5 [2022-10-26 14:53:18.551027] --log-level=user1:6 [2022-10-26 14:53:18.551031] --iova-mode=pa [2022-10-26 14:53:18.551035] --base-virtaddr=0x200000000000 [2022-10-26 14:53:18.551039] --match-allocations [2022-10-26 14:53:18.551043] --file-prefix=spdk_pid4266 [2022-10-26 14:53:18.551048] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:53:18.672613] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:53:18.707088] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:53:18.707639] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 283493.26 IOPS 138.42 MiB/s
=============================================================
Total : 283493.26 IOPS 138.42 MiB/s
[2022-10-26 14:53:31.638209] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:53:31.638256] [ DPDK EAL parameters: [2022-10-26 14:53:31.638262] bdevperf [2022-10-26 14:53:31.638266] --no-shconf [2022-10-26 14:53:31.638271] -l 0 [2022-10-26 14:53:31.638276] --huge-unlink [2022-10-26 14:53:31.638280] --log-level=lib.eal:6 [2022-10-26 14:53:31.638286] --log-level=lib.cryptodev:5 [2022-10-26 14:53:31.638290] --log-level=user1:6 [2022-10-26 14:53:31.638294] --iova-mode=pa [2022-10-26 14:53:31.638299] --base-virtaddr=0x200000000000 [2022-10-26 14:53:31.638304] --match-allocations [2022-10-26 14:53:31.638308] --file-prefix=spdk_pid4291 [2022-10-26 14:53:31.638313] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:53:31.760182] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:53:31.803079] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:53:31.803648] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 364445.67 IOPS 177.95 MiB/s
=============================================================
Total : 364445.67 IOPS 177.95 MiB/s
[2022-10-26 14:53:44.673297] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:53:44.673344] [ DPDK EAL parameters: [2022-10-26 14:53:44.673349] bdevperf [2022-10-26 14:53:44.673354] --no-shconf [2022-10-26 14:53:44.673358] -l 0 [2022-10-26 14:53:44.673363] --huge-unlink [2022-10-26 14:53:44.673368] --log-level=lib.eal:6 [2022-10-26 14:53:44.673373] --log-level=lib.cryptodev:5 [2022-10-26 14:53:44.673377] --log-level=user1:6 [2022-10-26 14:53:44.673381] --iova-mode=pa [2022-10-26 14:53:44.673386] --base-virtaddr=0x200000000000 [2022-10-26 14:53:44.673392] --match-allocations [2022-10-26 14:53:44.673396] --file-prefix=spdk_pid4316 [2022-10-26 14:53:44.673401] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:53:44.795838] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:53:44.839049] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:53:44.839650] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 366439.44 IOPS 178.93 MiB/s
=============================================================
Total : 366439.44 IOPS 178.93 MiB/s
[2022-10-26 14:53:57.645285] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:53:57.645331] [ DPDK EAL parameters: [2022-10-26 14:53:57.645337] bdevperf [2022-10-26 14:53:57.645342] --no-shconf [2022-10-26 14:53:57.645347] -l 0 [2022-10-26 14:53:57.645352] --huge-unlink [2022-10-26 14:53:57.645356] --log-level=lib.eal:6 [2022-10-26 14:53:57.645361] --log-level=lib.cryptodev:5 [2022-10-26 14:53:57.645366] --log-level=user1:6 [2022-10-26 14:53:57.645371] --iova-mode=pa [2022-10-26 14:53:57.645376] --base-virtaddr=0x200000000000 [2022-10-26 14:53:57.645381] --match-allocations [2022-10-26 14:53:57.645385] --file-prefix=spdk_pid4342 [2022-10-26 14:53:57.645390] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:53:57.767573] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:53:57.807080] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:53:57.807678] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 363187.05 IOPS 177.34 MiB/s
=============================================================
Total : 363187.05 IOPS 177.34 MiB/s
[2022-10-26 14:54:10.638786] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:54:10.638833] [ DPDK EAL parameters: [2022-10-26 14:54:10.638840] bdevperf [2022-10-26 14:54:10.638845] --no-shconf [2022-10-26 14:54:10.638850] -l 0 [2022-10-26 14:54:10.638854] --huge-unlink [2022-10-26 14:54:10.638858] --log-level=lib.eal:6 [2022-10-26 14:54:10.638862] --log-level=lib.cryptodev:5 [2022-10-26 14:54:10.638866] --log-level=user1:6 [2022-10-26 14:54:10.638871] --iova-mode=pa [2022-10-26 14:54:10.638876] --base-virtaddr=0x200000000000 [2022-10-26 14:54:10.638880] --match-allocations [2022-10-26 14:54:10.638884] --file-prefix=spdk_pid4368 [2022-10-26 14:54:10.638888] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:54:10.760857] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:54:10.799163] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:54:10.799767] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 384959.50 IOPS 187.97 MiB/s
=============================================================
Total : 384959.50 IOPS 187.97 MiB/s
[2022-10-26 14:54:23.653940] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:54:23.653988] [ DPDK EAL parameters: [2022-10-26 14:54:23.653994] bdevperf [2022-10-26 14:54:23.653998] --no-shconf [2022-10-26 14:54:23.654003] -l 0 [2022-10-26 14:54:23.654007] --huge-unlink [2022-10-26 14:54:23.654012] --log-level=lib.eal:6 [2022-10-26 14:54:23.654017] --log-level=lib.cryptodev:5 [2022-10-26 14:54:23.654022] --log-level=user1:6 [2022-10-26 14:54:23.654026] --iova-mode=pa [2022-10-26 14:54:23.654030] --base-virtaddr=0x200000000000 [2022-10-26 14:54:23.654035] --match-allocations [2022-10-26 14:54:23.654040] --file-prefix=spdk_pid4393 [2022-10-26 14:54:23.654045] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:54:23.776052] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:54:23.828570] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:54:23.829162] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 384611.58 IOPS 187.80 MiB/s
=============================================================
Total : 384611.58 IOPS 187.80 MiB/s
[2022-10-26 14:54:36.787399] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:54:36.787459] [ DPDK EAL parameters: [2022-10-26 14:54:36.787467] bdevperf [2022-10-26 14:54:36.787474] --no-shconf [2022-10-26 14:54:36.787481] -l 0 [2022-10-26 14:54:36.787486] --huge-unlink [2022-10-26 14:54:36.787491] --log-level=lib.eal:6 [2022-10-26 14:54:36.787497] --log-level=lib.cryptodev:5 [2022-10-26 14:54:36.787503] --log-level=user1:6 [2022-10-26 14:54:36.787510] --iova-mode=pa [2022-10-26 14:54:36.787516] --base-virtaddr=0x200000000000 [2022-10-26 14:54:36.787523] --match-allocations [2022-10-26 14:54:36.787530] --file-prefix=spdk_pid4418 [2022-10-26 14:54:36.787537] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:54:36.909401] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:54:36.951018] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:54:36.951625] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 387768.06 IOPS 189.34 MiB/s
=============================================================
Total : 387768.06 IOPS 189.34 MiB/s
[2022-10-26 14:54:49.793371] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:54:49.793413] [ DPDK EAL parameters: [2022-10-26 14:54:49.793420] bdevperf [2022-10-26 14:54:49.793424] --no-shconf [2022-10-26 14:54:49.793429] -l 0 [2022-10-26 14:54:49.793434] --huge-unlink [2022-10-26 14:54:49.793438] --log-level=lib.eal:6 [2022-10-26 14:54:49.793444] --log-level=lib.cryptodev:5 [2022-10-26 14:54:49.793448] --log-level=user1:6 [2022-10-26 14:54:49.793452] --iova-mode=pa [2022-10-26 14:54:49.793456] --base-virtaddr=0x200000000000 [2022-10-26 14:54:49.793460] --match-allocations [2022-10-26 14:54:49.793466] --file-prefix=spdk_pid4444 [2022-10-26 14:54:49.793470] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:54:49.915385] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:54:49.955076] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:54:49.955675] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 336723.59 IOPS 164.42 MiB/s
=============================================================
Total : 336723.59 IOPS 164.42 MiB/s
[2022-10-26 14:55:02.826118] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:55:02.826165] [ DPDK EAL parameters: [2022-10-26 14:55:02.826170] bdevperf [2022-10-26 14:55:02.826176] --no-shconf [2022-10-26 14:55:02.826180] -l 0 [2022-10-26 14:55:02.826185] --huge-unlink [2022-10-26 14:55:02.826189] --log-level=lib.eal:6 [2022-10-26 14:55:02.826194] --log-level=lib.cryptodev:5 [2022-10-26 14:55:02.826198] --log-level=user1:6 [2022-10-26 14:55:02.826202] --iova-mode=pa [2022-10-26 14:55:02.826206] --base-virtaddr=0x200000000000 [2022-10-26 14:55:02.826210] --match-allocations [2022-10-26 14:55:02.826215] --file-prefix=spdk_pid4469 [2022-10-26 14:55:02.826220] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:55:02.948254] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:55:02.990987] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:55:02.991582] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 338676.43 IOPS 165.37 MiB/s
=============================================================
Total : 338676.43 IOPS 165.37 MiB/s
[2022-10-26 14:55:15.880904] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:55:15.880951] [ DPDK EAL parameters: [2022-10-26 14:55:15.880957] bdevperf [2022-10-26 14:55:15.880962] --no-shconf [2022-10-26 14:55:15.880966] -l 0 [2022-10-26 14:55:15.880971] --huge-unlink [2022-10-26 14:55:15.880976] --log-level=lib.eal:6 [2022-10-26 14:55:15.880981] --log-level=lib.cryptodev:5 [2022-10-26 14:55:15.880985] --log-level=user1:6 [2022-10-26 14:55:15.880989] --iova-mode=pa [2022-10-26 14:55:15.880995] --base-virtaddr=0x200000000000 [2022-10-26 14:55:15.881000] --match-allocations [2022-10-26 14:55:15.881004] --file-prefix=spdk_pid4494 [2022-10-26 14:55:15.881009] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:55:16.003039] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:55:16.042971] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:55:16.043568] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 337224.20 IOPS 164.66 MiB/s
=============================================================
Total : 337224.20 IOPS 164.66 MiB/s
[2022-10-26 14:55:29.004977] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:55:29.005022] [ DPDK EAL parameters: [2022-10-26 14:55:29.005028] bdevperf [2022-10-26 14:55:29.005033] --no-shconf [2022-10-26 14:55:29.005037] -l 0 [2022-10-26 14:55:29.005042] --huge-unlink [2022-10-26 14:55:29.005046] --log-level=lib.eal:6 [2022-10-26 14:55:29.005051] --log-level=lib.cryptodev:5 [2022-10-26 14:55:29.005055] --log-level=user1:6 [2022-10-26 14:55:29.005060] --iova-mode=pa [2022-10-26 14:55:29.005064] --base-virtaddr=0x200000000000 [2022-10-26 14:55:29.005068] --match-allocations [2022-10-26 14:55:29.005072] --file-prefix=spdk_pid4520 [2022-10-26 14:55:29.005077] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:55:29.127554] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:55:29.171155] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:55:29.171762] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 559323.89 IOPS 273.11 MiB/s
=============================================================
Total : 559323.89 IOPS 273.11 MiB/s
[2022-10-26 14:55:42.157720] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:55:42.157764] [ DPDK EAL parameters: [2022-10-26 14:55:42.157769] bdevperf [2022-10-26 14:55:42.157774] --no-shconf [2022-10-26 14:55:42.157779] -l 0 [2022-10-26 14:55:42.157784] --huge-unlink [2022-10-26 14:55:42.157788] --log-level=lib.eal:6 [2022-10-26 14:55:42.157792] --log-level=lib.cryptodev:5 [2022-10-26 14:55:42.157795] --log-level=user1:6 [2022-10-26 14:55:42.157800] --iova-mode=pa [2022-10-26 14:55:42.157804] --base-virtaddr=0x200000000000 [2022-10-26 14:55:42.157809] --match-allocations [2022-10-26 14:55:42.157813] --file-prefix=spdk_pid4546 [2022-10-26 14:55:42.157818] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:55:42.279857] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:55:42.323115] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:55:42.323737] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 629634.72 IOPS 307.44 MiB/s
=============================================================
Total : 629634.72 IOPS 307.44 MiB/s
[2022-10-26 14:55:55.451763] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:55:55.451810] [ DPDK EAL parameters: [2022-10-26 14:55:55.451816] bdevperf [2022-10-26 14:55:55.451821] --no-shconf [2022-10-26 14:55:55.451826] -l 0 [2022-10-26 14:55:55.451831] --huge-unlink [2022-10-26 14:55:55.451836] --log-level=lib.eal:6 [2022-10-26 14:55:55.451841] --log-level=lib.cryptodev:5 [2022-10-26 14:55:55.451845] --log-level=user1:6 [2022-10-26 14:55:55.451849] --iova-mode=pa [2022-10-26 14:55:55.451854] --base-virtaddr=0x200000000000 [2022-10-26 14:55:55.451859] --match-allocations [2022-10-26 14:55:55.451863] --file-prefix=spdk_pid4572 [2022-10-26 14:55:55.451868] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:55:55.573975] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:55:55.607138] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:55:55.607755] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 506563.20 IOPS 247.35 MiB/s
=============================================================
Total : 506563.20 IOPS 247.35 MiB/s
[2022-10-26 14:56:08.514571] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:56:08.514615] [ DPDK EAL parameters: [2022-10-26 14:56:08.514621] bdevperf [2022-10-26 14:56:08.514626] --no-shconf [2022-10-26 14:56:08.514631] -l 0 [2022-10-26 14:56:08.514636] --huge-unlink [2022-10-26 14:56:08.514641] --log-level=lib.eal:6 [2022-10-26 14:56:08.514645] --log-level=lib.cryptodev:5 [2022-10-26 14:56:08.514649] --log-level=user1:6 [2022-10-26 14:56:08.514653] --iova-mode=pa [2022-10-26 14:56:08.514657] --base-virtaddr=0x200000000000 [2022-10-26 14:56:08.514661] --match-allocations [2022-10-26 14:56:08.514667] --file-prefix=spdk_pid4598 [2022-10-26 14:56:08.514672] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:56:08.636693] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:56:08.679060] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:56:08.679646] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 482815.98 IOPS 235.75 MiB/s
=============================================================
Total : 482815.98 IOPS 235.75 MiB/s
[2022-10-26 14:56:21.518478] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:56:21.518523] [ DPDK EAL parameters: [2022-10-26 14:56:21.518529] bdevperf [2022-10-26 14:56:21.518534] --no-shconf [2022-10-26 14:56:21.518538] -l 0 [2022-10-26 14:56:21.518543] --huge-unlink [2022-10-26 14:56:21.518547] --log-level=lib.eal:6 [2022-10-26 14:56:21.518551] --log-level=lib.cryptodev:5 [2022-10-26 14:56:21.518556] --log-level=user1:6 [2022-10-26 14:56:21.518560] --iova-mode=pa [2022-10-26 14:56:21.518564] --base-virtaddr=0x200000000000 [2022-10-26 14:56:21.518568] --match-allocations [2022-10-26 14:56:21.518575] --file-prefix=spdk_pid4623 [2022-10-26 14:56:21.518580] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:56:21.640578] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:56:21.679004] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:56:21.679588] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
Steps uses worklets. A description of the worklet used by
the current step is provided below. The description is extracted from the docstring
of the worklet and provided here in verbatim / as-is.
Extract and transform fio and bdevperf output
=============================================
fio has:
* bandwidth in KiB / second
* latency is mean and in nano-seconds
* IOPS are 'raw' / base-unit
bdevperf has:
* bandwidth in MiB / second
* IOPS are 'raw' / base-unit
NOTE: The metric-context need addition of backend-options, the options are
semi-encoded by ioengine and 'label', however, that is not very precise.
{'description': '\n'
' Extract and transform fio and bdevperf output\n'
' =============================================\n'
'\n'
' fio has:\n'
'\n'
' * bandwidth in KiB / second\n'
' * latency is mean and in nano-seconds\n'
" * IOPS are 'raw' / base-unit\n"
'\n'
' bdevperf has:\n'
'\n'
' * bandwidth in MiB / second\n'
" * IOPS are 'raw' / base-unit\n"
'\n'
' NOTE: The metric-context need addition of '
'backend-options, the options are\n'
" semi-encoded by ioengine and 'label', however, that is "
'not very precise.\n',
'extras': {},
'id': '002_plot_bdevperf',
'name': 'plot_bdevperf',
'nr': 2,
'status': {'elapsed': 0.706589937210083,
'failed': 0,
'passed': 1,
'skipped': 0,
'started': 0.0},
'uses': 'plot',
'with': {'path': 'cijoe-output',
'plot_attributes': 'plot-attributes-512.yaml',
'tool': 'bdevperf'}}
Steps uses worklets. A description of the worklet used by
the current step is provided below. The description is extracted from the docstring
of the worklet and provided here in verbatim / as-is.
Run fio...
{'description': '\n Run fio...\n',
'extras': {'runlog': {'cmd_01': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_01.output'),
'state': {'begin': 1666782046.3969247,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8351426124572754,
'end': 1666782047.2320673,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_01.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_01.state')},
'cmd_02': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_02.output'),
'state': {'begin': 1666782047.2342446,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.069629430770874,
'end': 1666782058.303874,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_02.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_02.state')},
'cmd_03': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789148,\n'
' "timestamp_ms" : '
'1666789148400,\n'
' "time" : "Wed Oct 26 14:59:08 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 394994176,\n'
' "io_kbytes" : 385736,\n'
' "bw_bytes" : 56419679,\n'
' "bw" : 55097,\n'
' "iops" : 110194.686473,\n'
' "runtime" : 7001,\n'
' "total_ios" : 771473,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1362,\n'
' "max" : 5961,\n'
' "mean" : 1460.212779,\n'
' "stddev" : 51.214525,\n'
' "N" : 771473\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2955,\n'
' "max" : 117891,\n'
' "mean" : 7271.748888,\n'
' "stddev" : 971.362791,\n'
' "N" : 771473,\n'
' "percentile" : {\n'
' "1.000000" : 6752,\n'
' "5.000000" : 7008,\n'
' "10.000000" : 7008,\n'
' "20.000000" : 7072,\n'
' "30.000000" : 7072,\n'
' "40.000000" : 7136,\n'
' "50.000000" : 7200,\n'
' "60.000000" : 7200,\n'
' "70.000000" : 7264,\n'
' "80.000000" : 7264,\n'
' "90.000000" : 7392,\n'
' "95.000000" : 7456,\n'
' "99.000000" : 10432,\n'
' "99.500000" : 13632,\n'
' "99.900000" : 15680,\n'
' "99.950000" : 15808,\n'
' "99.990000" : 57600\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7354,\n'
' "max" : 119324,\n'
' "mean" : 8731.961667,\n'
' "stddev" : 973.813190,\n'
' "N" : 771473\n'
' },\n'
' "bw_min" : 54956,\n'
' "bw_max" : 55331,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'55107.071429,\n'
' "bw_dev" : 106.053299,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 109912,\n'
' "iops_max" : 110662,\n'
' "iops_mean" : '
'110214.142857,\n'
' "iops_stddev" : '
'212.106598,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 8.028571,\n'
' "sys_cpu" : 35.128571,\n'
' "ctx" : 771473,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 98.671243,\n'
' "20" : 1.310610,\n'
' "50" : 0.010000,\n'
' "100" : 0.012703,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=4935: Wed Oct 26 14:59:08 '
'2022\n'
' read: IOPS=110k, BW=53.8MiB/s '
'(56.4MB/s)(377MiB/7001msec)\n'
' slat (nsec): min=1362, '
'max=5961, avg=1460.21, '
'stdev=51.21\n'
' clat (nsec): min=2955, '
'max=117891, avg=7271.75, '
'stdev=971.36\n'
' lat (nsec): min=7354, '
'max=119324, avg=8731.96, '
'stdev=973.81\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6752], 5.00th=[ '
'7008], 10.00th=[ 7008], 20.00th=[ '
'7072],\n'
' | 30.00th=[ 7072], 40.00th=[ '
'7136], 50.00th=[ 7200], 60.00th=[ '
'7200],\n'
' | 70.00th=[ 7264], 80.00th=[ '
'7264], 90.00th=[ 7392], 95.00th=[ '
'7456],\n'
' | 99.00th=[10432], '
'99.50th=[13632], 99.90th=[15680], '
'99.95th=[15808],\n'
' | 99.99th=[57600]\n'
' bw ( KiB/s): min=54956, '
'max=55331, per=100.00%, '
'avg=55107.07, stdev=106.05, '
'samples=14\n'
' iops : min=109912, '
'max=110662, avg=110214.14, '
'stdev=212.11, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=98.67%, 20=1.31%, 50=0.01%, '
'100=0.01%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=8.03%, '
'sys=35.13%, ctx=771473, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=771473,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=53.8MiB/s (56.4MB/s), '
'53.8MiB/s-53.8MiB/s '
'(56.4MB/s-56.4MB/s), io=377MiB '
'(395MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_03.output'),
'state': {'begin': 1666782058.306314,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8316667079925537,
'end': 1666782059.1379807,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_03.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_03.state')},
'cmd_04': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_04.output'),
'state': {'begin': 1666782060.2719457,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8387961387634277,
'end': 1666782061.1107419,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_04.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_04.state')},
'cmd_05': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_05.output'),
'state': {'begin': 1666782061.1129358,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.059051752090454,
'end': 1666782072.1719875,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_05.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_05.state')},
'cmd_06': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789162,\n'
' "timestamp_ms" : '
'1666789162269,\n'
' "time" : "Wed Oct 26 14:59:22 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 394686464,\n'
' "io_kbytes" : 385436,\n'
' "bw_bytes" : 56383780,\n'
' "bw" : 55062,\n'
' "iops" : 110124.571429,\n'
' "runtime" : 7000,\n'
' "total_ios" : 770872,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1402,\n'
' "max" : 13765,\n'
' "mean" : 1493.597215,\n'
' "stddev" : 57.341593,\n'
' "N" : 770872\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 171,\n'
' "max" : 77646,\n'
' "mean" : 7265.220271,\n'
' "stddev" : 980.277945,\n'
' "N" : 770872,\n'
' "percentile" : {\n'
' "1.000000" : 6752,\n'
' "5.000000" : 7008,\n'
' "10.000000" : 7008,\n'
' "20.000000" : 7072,\n'
' "30.000000" : 7072,\n'
' "40.000000" : 7136,\n'
' "50.000000" : 7200,\n'
' "60.000000" : 7200,\n'
' "70.000000" : 7264,\n'
' "80.000000" : 7264,\n'
' "90.000000" : 7392,\n'
' "95.000000" : 7456,\n'
' "99.000000" : 10560,\n'
' "99.500000" : 13504,\n'
' "99.900000" : 15680,\n'
' "99.950000" : 15808,\n'
' "99.990000" : 59136\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7434,\n'
' "max" : 79108,\n'
' "mean" : 8758.817486,\n'
' "stddev" : 983.505773,\n'
' "N" : 770872\n'
' },\n'
' "bw_min" : 54999,\n'
' "bw_max" : 55216,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'55079.642857,\n'
' "bw_dev" : 51.880939,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 109998,\n'
' "iops_max" : 110432,\n'
' "iops_mean" : '
'110159.285714,\n'
' "iops_stddev" : '
'103.761879,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 6.757143,\n'
' "sys_cpu" : 36.657143,\n'
' "ctx" : 770872,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 98.658921,\n'
' "20" : 1.323436,\n'
' "50" : 0.010000,\n'
' "100" : 0.013751,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=4981: Wed Oct 26 14:59:22 '
'2022\n'
' read: IOPS=110k, BW=53.8MiB/s '
'(56.4MB/s)(376MiB/7000msec)\n'
' slat (nsec): min=1402, '
'max=13765, avg=1493.60, '
'stdev=57.34\n'
' clat (nsec): min=171, '
'max=77646, avg=7265.22, '
'stdev=980.28\n'
' lat (nsec): min=7434, '
'max=79108, avg=8758.82, '
'stdev=983.51\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6752], 5.00th=[ '
'7008], 10.00th=[ 7008], 20.00th=[ '
'7072],\n'
' | 30.00th=[ 7072], 40.00th=[ '
'7136], 50.00th=[ 7200], 60.00th=[ '
'7200],\n'
' | 70.00th=[ 7264], 80.00th=[ '
'7264], 90.00th=[ 7392], 95.00th=[ '
'7456],\n'
' | 99.00th=[10560], '
'99.50th=[13504], 99.90th=[15680], '
'99.95th=[15808],\n'
' | 99.99th=[59136]\n'
' bw ( KiB/s): min=54999, '
'max=55216, per=100.00%, '
'avg=55079.64, stdev=51.88, '
'samples=14\n'
' iops : min=109998, '
'max=110432, avg=110159.29, '
'stdev=103.76, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=98.66%, 20=1.32%, 50=0.01%, '
'100=0.01%\n'
' cpu : usr=6.76%, '
'sys=36.66%, ctx=770872, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=770872,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=53.8MiB/s (56.4MB/s), '
'53.8MiB/s-53.8MiB/s '
'(56.4MB/s-56.4MB/s), io=376MiB '
'(395MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_06.output'),
'state': {'begin': 1666782072.174585,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8430523872375488,
'end': 1666782073.0176375,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_06.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_06.state')},
'cmd_07': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_07.output'),
'state': {'begin': 1666782074.2136307,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7940611839294434,
'end': 1666782075.0076919,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_07.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_07.state')},
'cmd_08': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_08.output'),
'state': {'begin': 1666782075.0098498,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.038039922714233,
'end': 1666782086.0478897,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_08.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_08.state')},
'cmd_09': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789176,\n'
' "timestamp_ms" : '
'1666789176144,\n'
' "time" : "Wed Oct 26 14:59:36 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 397914112,\n'
' "io_kbytes" : 388588,\n'
' "bw_bytes" : 56836753,\n'
' "bw" : 55504,\n'
' "iops" : 111009.141551,\n'
' "runtime" : 7001,\n'
' "total_ios" : 777175,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1352,\n'
' "max" : 16331,\n'
' "mean" : 1435.588716,\n'
' "stddev" : 64.972788,\n'
' "N" : 777176\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 200,\n'
' "max" : 118251,\n'
' "mean" : 7248.129245,\n'
' "stddev" : '
'1059.290764,\n'
' "N" : 777175,\n'
' "percentile" : {\n'
' "1.000000" : 6752,\n'
' "5.000000" : 6944,\n'
' "10.000000" : 7008,\n'
' "20.000000" : 7008,\n'
' "30.000000" : 7072,\n'
' "40.000000" : 7072,\n'
' "50.000000" : 7136,\n'
' "60.000000" : 7200,\n'
' "70.000000" : 7200,\n'
' "80.000000" : 7264,\n'
' "90.000000" : 7328,\n'
' "95.000000" : 7456,\n'
' "99.000000" : 10688,\n'
' "99.500000" : 13632,\n'
' "99.900000" : 15680,\n'
' "99.950000" : 15936,\n'
' "99.990000" : 60672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7274,\n'
' "max" : 119724,\n'
' "mean" : 8683.718016,\n'
' "stddev" : '
'1062.030108,\n'
' "N" : 777175\n'
' },\n'
' "bw_min" : 55395,\n'
' "bw_max" : 55748,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'55535.571429,\n'
' "bw_dev" : 121.563984,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 110790,\n'
' "iops_max" : 111496,\n'
' "iops_mean" : '
'111071.285714,\n'
' "iops_stddev" : '
'243.041443,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 7.557143,\n'
' "sys_cpu" : 35.600000,\n'
' "ctx" : 777179,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 98.510245,\n'
' "20" : 1.465307,\n'
' "50" : 0.010000,\n'
' "100" : 0.017757,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=5027: Wed Oct 26 14:59:36 '
'2022\n'
' read: IOPS=111k, BW=54.2MiB/s '
'(56.8MB/s)(379MiB/7001msec)\n'
' slat (nsec): min=1352, '
'max=16331, avg=1435.59, '
'stdev=64.97\n'
' clat (nsec): min=200, '
'max=118251, avg=7248.13, '
'stdev=1059.29\n'
' lat (usec): min=7, max=119, '
'avg= 8.68, stdev= 1.06\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6752], 5.00th=[ '
'6944], 10.00th=[ 7008], 20.00th=[ '
'7008],\n'
' | 30.00th=[ 7072], 40.00th=[ '
'7072], 50.00th=[ 7136], 60.00th=[ '
'7200],\n'
' | 70.00th=[ 7200], 80.00th=[ '
'7264], 90.00th=[ 7328], 95.00th=[ '
'7456],\n'
' | 99.00th=[10688], '
'99.50th=[13632], 99.90th=[15680], '
'99.95th=[15936],\n'
' | 99.99th=[60672]\n'
' bw ( KiB/s): min=55395, '
'max=55748, per=100.00%, '
'avg=55535.57, stdev=121.56, '
'samples=14\n'
' iops : min=110790, '
'max=111496, avg=111071.29, '
'stdev=243.04, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=98.51%, 20=1.47%, '
'50=0.01%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=7.56%, '
'sys=35.60%, ctx=777179, majf=0, '
'minf=1\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=777175,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=54.2MiB/s (56.8MB/s), '
'54.2MiB/s-54.2MiB/s '
'(56.8MB/s-56.8MB/s), io=379MiB '
'(398MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_09.output'),
'state': {'begin': 1666782086.0503554,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.8288614749908447,
'end': 1666782086.879217,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_09.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_09.state')},
'cmd_10': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_10.output'),
'state': {'begin': 1666782087.9278235,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7727639675140381,
'end': 1666782088.7005875,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_10.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_10.state')},
'cmd_100': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_100.output'),
'state': {'begin': 1666782525.8023028,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7964434623718262,
'end': 1666782526.5987463,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_100.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_100.state')},
'cmd_101': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_101.output'),
'state': {'begin': 1666782526.6009583,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.07068920135498,
'end': 1666782537.6716475,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_101.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_101.state')},
'cmd_102': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789627,\n'
' "timestamp_ms" : '
'1666789627760,\n'
' "time" : "Wed Oct 26 15:07:07 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 762248704,\n'
' "io_kbytes" : 744383,\n'
' "bw_bytes" : 108877118,\n'
' "bw" : 106325,\n'
' "iops" : 212650.478503,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1488766,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1172,\n'
' "max" : 21300,\n'
' "mean" : 1340.914016,\n'
' "stddev" : '
'138.459284,\n'
' "N" : 1488766\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 151,\n'
' "max" : 118081,\n'
' "mean" : 7781.440333,\n'
' "stddev" : '
'2292.078431,\n'
' "N" : 1488767,\n'
' "percentile" : {\n'
' "1.000000" : 5664,\n'
' "5.000000" : 5792,\n'
' "10.000000" : 5792,\n'
' "20.000000" : 5856,\n'
' "30.000000" : 5856,\n'
' "40.000000" : 5920,\n'
' "50.000000" : 9024,\n'
' "60.000000" : 9152,\n'
' "70.000000" : 9280,\n'
' "80.000000" : 9280,\n'
' "90.000000" : 9408,\n'
' "95.000000" : 9536,\n'
' "99.000000" : '
'16768,\n'
' "99.500000" : '
'18048,\n'
' "99.900000" : '
'19072,\n'
' "99.950000" : '
'23936,\n'
' "99.990000" : 64768\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5661,\n'
' "max" : 119274,\n'
' "mean" : 9122.354383,\n'
' "stddev" : '
'2373.673269,\n'
' "N" : 1488767\n'
' },\n'
' "bw_min" : 106162,\n'
' "bw_max" : 106787,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'106405.307692,\n'
' "bw_dev" : 184.502206,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 212324,\n'
' "iops_max" : 213575,\n'
' "iops_mean" : '
'212810.769231,\n'
' "iops_stddev" : '
'369.150185,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.371429,\n'
' "sys_cpu" : 46.300000,\n'
' "ctx" : 746929,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.396434,\n'
' "20" : 2.514230,\n'
' "50" : 0.056221,\n'
' "100" : 0.025928,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=6649: Wed Oct 26 15:07:07 '
'2022\n'
' read: IOPS=213k, BW=104MiB/s '
'(109MB/s)(727MiB/7001msec)\n'
' slat (nsec): min=1172, '
'max=21300, avg=1340.91, '
'stdev=138.46\n'
' clat (nsec): min=151, '
'max=118081, avg=7781.44, '
'stdev=2292.08\n'
' lat (usec): min=5, max=119, '
'avg= 9.12, stdev= 2.37\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5664], '
'5.00th=[ 5792], 10.00th=[ 5792], '
'20.00th=[ 5856],\n'
' | 30.00th=[ 5856], '
'40.00th=[ 5920], 50.00th=[ '
'9024], 60.00th=[ 9152],\n'
' | 70.00th=[ 9280], '
'80.00th=[ 9280], 90.00th=[ '
'9408], 95.00th=[ 9536],\n'
' | 99.00th=[16768], '
'99.50th=[18048], '
'99.90th=[19072], '
'99.95th=[23936],\n'
' | 99.99th=[64768]\n'
' bw ( KiB/s): min=106162, '
'max=106787, per=100.00%, '
'avg=106405.31, stdev=184.50, '
'samples=13\n'
' iops : min=212324, '
'max=213575, avg=212810.77, '
'stdev=369.15, samples=13\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=97.40%, 20=2.51%, 50=0.06%, '
'100=0.03%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=14.37%, '
'sys=46.30%, ctx=746929, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1488766,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=104MiB/s (109MB/s), '
'104MiB/s-104MiB/s '
'(109MB/s-109MB/s), io=727MiB '
'(762MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_102.output'),
'state': {'begin': 1666782537.6741395,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8083879947662354,
'end': 1666782538.4825275,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_102.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_102.state')},
'cmd_103': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_103.output'),
'state': {'begin': 1666782539.5181973,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8535711765289307,
'end': 1666782540.3717685,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_103.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_103.state')},
'cmd_104': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_104.output'),
'state': {'begin': 1666782540.373811,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.071117401123047,
'end': 1666782551.4449284,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_104.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_104.state')},
'cmd_105': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789641,\n'
' "timestamp_ms" : '
'1666789641533,\n'
' "time" : "Wed Oct 26 15:07:21 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 763893760,\n'
' "io_kbytes" : 745990,\n'
' "bw_bytes" : 109112092,\n'
' "bw" : 106554,\n'
' "iops" : 213109.270104,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1491978,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1162,\n'
' "max" : 46527,\n'
' "mean" : 1340.729218,\n'
' "stddev" : '
'143.970424,\n'
' "N" : 1491979\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 150,\n'
' "max" : 118172,\n'
' "mean" : 7760.151302,\n'
' "stddev" : '
'2264.115071,\n'
' "N" : 1491979,\n'
' "percentile" : {\n'
' "1.000000" : 5600,\n'
' "5.000000" : 5792,\n'
' "10.000000" : 5792,\n'
' "20.000000" : 5856,\n'
' "30.000000" : 5856,\n'
' "40.000000" : 5920,\n'
' "50.000000" : 8896,\n'
' "60.000000" : 9152,\n'
' "70.000000" : 9280,\n'
' "80.000000" : 9280,\n'
' "90.000000" : 9408,\n'
' "95.000000" : 9536,\n'
' "99.000000" : '
'16768,\n'
' "99.500000" : '
'18048,\n'
' "99.900000" : '
'18816,\n'
' "99.950000" : '
'23680,\n'
' "99.990000" : 60672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5590,\n'
' "max" : 119384,\n'
' "mean" : 9100.880520,\n'
' "stddev" : '
'2350.258010,\n'
' "N" : 1491979\n'
' },\n'
' "bw_min" : 106122,\n'
' "bw_max" : 106923,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'106573.500000,\n'
' "bw_dev" : 240.951559,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 212244,\n'
' "iops_max" : 213846,\n'
' "iops_mean" : '
'213147.000000,\n'
' "iops_stddev" : '
'481.903118,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.057143,\n'
' "sys_cpu" : 46.657143,\n'
' "ctx" : 748714,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.423286,\n'
' "20" : 2.503053,\n'
' "50" : 0.046985,\n'
' "100" : 0.020979,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=6695: Wed Oct 26 15:07:21 '
'2022\n'
' read: IOPS=213k, BW=104MiB/s '
'(109MB/s)(729MiB/7001msec)\n'
' slat (nsec): min=1162, '
'max=46527, avg=1340.73, '
'stdev=143.97\n'
' clat (nsec): min=150, '
'max=118172, avg=7760.15, '
'stdev=2264.12\n'
' lat (usec): min=5, max=119, '
'avg= 9.10, stdev= 2.35\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5600], '
'5.00th=[ 5792], 10.00th=[ 5792], '
'20.00th=[ 5856],\n'
' | 30.00th=[ 5856], '
'40.00th=[ 5920], 50.00th=[ '
'8896], 60.00th=[ 9152],\n'
' | 70.00th=[ 9280], '
'80.00th=[ 9280], 90.00th=[ '
'9408], 95.00th=[ 9536],\n'
' | 99.00th=[16768], '
'99.50th=[18048], '
'99.90th=[18816], '
'99.95th=[23680],\n'
' | 99.99th=[60672]\n'
' bw ( KiB/s): min=106122, '
'max=106923, per=100.00%, '
'avg=106573.50, stdev=240.95, '
'samples=14\n'
' iops : min=212244, '
'max=213846, avg=213147.00, '
'stdev=481.90, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=97.42%, 20=2.50%, 50=0.05%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=14.06%, '
'sys=46.66%, ctx=748714, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1491978,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=104MiB/s (109MB/s), '
'104MiB/s-104MiB/s '
'(109MB/s-109MB/s), io=729MiB '
'(764MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_105.output'),
'state': {'begin': 1666782551.4476588,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8775582313537598,
'end': 1666782552.325217,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_105.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_105.state')},
'cmd_106': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_106.output'),
'state': {'begin': 1666782553.3838997,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7833948135375977,
'end': 1666782554.1672945,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_106.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_106.state')},
'cmd_107': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_107.output'),
'state': {'begin': 1666782554.1695273,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.074486255645752,
'end': 1666782565.2440135,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_107.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_107.state')},
'cmd_108': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789655,\n'
' "timestamp_ms" : '
'1666789655332,\n'
' "time" : "Wed Oct 26 15:07:35 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 764250112,\n'
' "io_kbytes" : 746338,\n'
' "bw_bytes" : 109162992,\n'
' "bw" : 106604,\n'
' "iops" : 213208.827310,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1492675,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1162,\n'
' "max" : 25678,\n'
' "mean" : 1346.666306,\n'
' "stddev" : '
'139.012440,\n'
' "N" : 1492675\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 90,\n'
' "max" : 244338,\n'
' "mean" : 7752.355304,\n'
' "stddev" : '
'2295.470051,\n'
' "N" : 1492676,\n'
' "percentile" : {\n'
' "1.000000" : 5600,\n'
' "5.000000" : 5728,\n'
' "10.000000" : 5792,\n'
' "20.000000" : 5792,\n'
' "30.000000" : 5856,\n'
' "40.000000" : 5920,\n'
' "50.000000" : 9024,\n'
' "60.000000" : 9152,\n'
' "70.000000" : 9280,\n'
' "80.000000" : 9280,\n'
' "90.000000" : 9408,\n'
' "95.000000" : 9536,\n'
' "99.000000" : '
'16768,\n'
' "99.500000" : '
'18048,\n'
' "99.900000" : '
'18816,\n'
' "99.950000" : '
'23680,\n'
' "99.990000" : 63232\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5621,\n'
' "max" : 245951,\n'
' "mean" : 9099.021662,\n'
' "stddev" : '
'2381.626970,\n'
' "N" : 1492676\n'
' },\n'
' "bw_min" : 106186,\n'
' "bw_max" : 107335,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'106623.785714,\n'
' "bw_dev" : 356.455883,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 212372,\n'
' "iops_max" : 214671,\n'
' "iops_mean" : '
'213247.642857,\n'
' "iops_stddev" : '
'713.065280,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 12.200000,\n'
' "sys_cpu" : 48.442857,\n'
' "ctx" : 748942,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 97.413905,\n'
' "20" : 2.505435,\n'
' "50" : 0.049776,\n'
' "100" : 0.022912,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=6741: Wed Oct 26 15:07:35 '
'2022\n'
' read: IOPS=213k, BW=104MiB/s '
'(109MB/s)(729MiB/7001msec)\n'
' slat (nsec): min=1162, '
'max=25678, avg=1346.67, '
'stdev=139.01\n'
' clat (nsec): min=90, '
'max=244338, avg=7752.36, '
'stdev=2295.47\n'
' lat (usec): min=5, max=245, '
'avg= 9.10, stdev= 2.38\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5600], '
'5.00th=[ 5728], 10.00th=[ 5792], '
'20.00th=[ 5792],\n'
' | 30.00th=[ 5856], '
'40.00th=[ 5920], 50.00th=[ '
'9024], 60.00th=[ 9152],\n'
' | 70.00th=[ 9280], '
'80.00th=[ 9280], 90.00th=[ '
'9408], 95.00th=[ 9536],\n'
' | 99.00th=[16768], '
'99.50th=[18048], '
'99.90th=[18816], '
'99.95th=[23680],\n'
' | 99.99th=[63232]\n'
' bw ( KiB/s): min=106186, '
'max=107335, per=100.00%, '
'avg=106623.79, stdev=356.46, '
'samples=14\n'
' iops : min=212372, '
'max=214671, avg=213247.64, '
'stdev=713.07, samples=14\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=97.41%, 20=2.51%, '
'50=0.05%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=12.20%, '
'sys=48.44%, ctx=748942, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1492675,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=104MiB/s (109MB/s), '
'104MiB/s-104MiB/s '
'(109MB/s-109MB/s), io=729MiB '
'(764MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_108.output'),
'state': {'begin': 1666782565.2477517,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.838524341583252,
'end': 1666782566.086276,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_108.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_108.state')},
'cmd_109': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_109.output'),
'state': {'begin': 1666782567.152859,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8400983810424805,
'end': 1666782567.9929574,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_109.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_109.state')},
'cmd_11': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_11.output'),
'state': {'begin': 1666782088.7027469,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.09110140800476,
'end': 1666782099.7938483,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_11.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_11.state')},
'cmd_110': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_110.output'),
'state': {'begin': 1666782567.9955487,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.147419691085815,
'end': 1666782579.1429684,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_110.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_110.state')},
'cmd_111': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789669,\n'
' "timestamp_ms" : '
'1666789669194,\n'
' "time" : "Wed Oct 26 15:07:49 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 760012288,\n'
' "io_kbytes" : 742199,\n'
' "bw_bytes" : 108557675,\n'
' "bw" : 106013,\n'
' "iops" : 212026.424796,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1484397,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1222,\n'
' "max" : 22522,\n'
' "mean" : 1423.243689,\n'
' "stddev" : '
'173.920884,\n'
' "N" : 1484398\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 151,\n'
' "max" : 115626,\n'
' "mean" : 7748.459844,\n'
' "stddev" : '
'2130.869047,\n'
' "N" : 1484398,\n'
' "percentile" : {\n'
' "1.000000" : 5536,\n'
' "5.000000" : 5664,\n'
' "10.000000" : 5728,\n'
' "20.000000" : 5792,\n'
' "30.000000" : 6176,\n'
' "40.000000" : 7328,\n'
' "50.000000" : 7520,\n'
' "60.000000" : 7712,\n'
' "70.000000" : 9280,\n'
' "80.000000" : 9408,\n'
' "90.000000" : 9536,\n'
' "95.000000" : 9664,\n'
' "99.000000" : '
'16768,\n'
' "99.500000" : '
'18304,\n'
' "99.900000" : '
'23936,\n'
' "99.950000" : '
'24448,\n'
' "99.990000" : 61184\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5751,\n'
' "max" : 116909,\n'
' "mean" : 9171.703695,\n'
' "stddev" : '
'2188.738867,\n'
' "N" : 1484398\n'
' },\n'
' "bw_min" : 105599,\n'
' "bw_max" : 106438,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'106032.142857,\n'
' "bw_dev" : 187.439367,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 211198,\n'
' "iops_max" : 212876,\n'
' "iops_mean" : '
'212064.285714,\n'
' "iops_stddev" : '
'374.878735,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.985714,\n'
' "sys_cpu" : 46.657143,\n'
' "ctx" : 745370,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.303619,\n'
' "20" : 2.542110,\n'
' "50" : 0.122811,\n'
' "100" : 0.021288,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=6787: Wed '
'Oct 26 15:07:49 2022\n'
' read: IOPS=212k, BW=104MiB/s '
'(109MB/s)(725MiB/7001msec)\n'
' slat (nsec): min=1222, '
'max=22522, avg=1423.24, '
'stdev=173.92\n'
' clat (nsec): min=151, '
'max=115626, avg=7748.46, '
'stdev=2130.87\n'
' lat (usec): min=5, max=116, '
'avg= 9.17, stdev= 2.19\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5536], '
'5.00th=[ 5664], 10.00th=[ 5728], '
'20.00th=[ 5792],\n'
' | 30.00th=[ 6176], '
'40.00th=[ 7328], 50.00th=[ '
'7520], 60.00th=[ 7712],\n'
' | 70.00th=[ 9280], '
'80.00th=[ 9408], 90.00th=[ '
'9536], 95.00th=[ 9664],\n'
' | 99.00th=[16768], '
'99.50th=[18304], '
'99.90th=[23936], '
'99.95th=[24448],\n'
' | 99.99th=[61184]\n'
' bw ( KiB/s): min=105599, '
'max=106438, per=100.00%, '
'avg=106032.14, stdev=187.44, '
'samples=14\n'
' iops : min=211198, '
'max=212876, avg=212064.29, '
'stdev=374.88, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=97.30%, 20=2.54%, 50=0.12%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=15.99%, '
'sys=46.66%, ctx=745370, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1484397,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=104MiB/s (109MB/s), '
'104MiB/s-104MiB/s '
'(109MB/s-109MB/s), io=725MiB '
'(760MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_111.output'),
'state': {'begin': 1666782579.1459336,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8462259769439697,
'end': 1666782579.9921596,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_111.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_111.state')},
'cmd_112': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_112.output'),
'state': {'begin': 1666782581.058541,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8410971164703369,
'end': 1666782581.8996382,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_112.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_112.state')},
'cmd_113': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_113.output'),
'state': {'begin': 1666782581.9015534,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.12335753440857,
'end': 1666782593.024911,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_113.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_113.state')},
'cmd_114': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789683,\n'
' "timestamp_ms" : '
'1666789683112,\n'
' "time" : "Wed Oct 26 15:08:03 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 762302976,\n'
' "io_kbytes" : 744436,\n'
' "bw_bytes" : 108884870,\n'
' "bw" : 106332,\n'
' "iops" : 212665.619197,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1488872,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1222,\n'
' "max" : 13836,\n'
' "mean" : 1386.090252,\n'
' "stddev" : '
'141.744320,\n'
' "N" : 1488872\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1954,\n'
' "max" : 79479,\n'
' "mean" : 7760.864534,\n'
' "stddev" : '
'2225.390339,\n'
' "N" : 1488873,\n'
' "percentile" : {\n'
' "1.000000" : 5536,\n'
' "5.000000" : 5728,\n'
' "10.000000" : 5728,\n'
' "20.000000" : 5792,\n'
' "30.000000" : 5856,\n'
' "40.000000" : 6496,\n'
' "50.000000" : 7520,\n'
' "60.000000" : 9152,\n'
' "70.000000" : 9280,\n'
' "80.000000" : 9408,\n'
' "90.000000" : 9408,\n'
' "95.000000" : 9536,\n'
' "99.000000" : '
'16768,\n'
' "99.500000" : '
'18304,\n'
' "99.900000" : '
'22400,\n'
' "99.950000" : '
'24192,\n'
' "99.990000" : 61696\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5721,\n'
' "max" : 80751,\n'
' "mean" : 9146.954825,\n'
' "stddev" : '
'2289.506518,\n'
' "N" : 1488873\n'
' },\n'
' "bw_min" : 106032,\n'
' "bw_max" : 106788,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'106351.571429,\n'
' "bw_dev" : 183.468176,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 212064,\n'
' "iops_max" : 213577,\n'
' "iops_mean" : '
'212703.214286,\n'
' "iops_stddev" : '
'367.119385,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.785714,\n'
' "sys_cpu" : 46.285714,\n'
' "ctx" : 747296,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 97.373313,\n'
' "20" : 2.508879,\n'
' "50" : 0.086441,\n'
' "100" : 0.022433,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=6834: Wed '
'Oct 26 15:08:03 2022\n'
' read: IOPS=213k, BW=104MiB/s '
'(109MB/s)(727MiB/7001msec)\n'
' slat (nsec): min=1222, '
'max=13836, avg=1386.09, '
'stdev=141.74\n'
' clat (nsec): min=1954, '
'max=79479, avg=7760.86, '
'stdev=2225.39\n'
' lat (nsec): min=5721, '
'max=80751, avg=9146.95, '
'stdev=2289.51\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5536], '
'5.00th=[ 5728], 10.00th=[ 5728], '
'20.00th=[ 5792],\n'
' | 30.00th=[ 5856], '
'40.00th=[ 6496], 50.00th=[ '
'7520], 60.00th=[ 9152],\n'
' | 70.00th=[ 9280], '
'80.00th=[ 9408], 90.00th=[ '
'9408], 95.00th=[ 9536],\n'
' | 99.00th=[16768], '
'99.50th=[18304], '
'99.90th=[22400], '
'99.95th=[24192],\n'
' | 99.99th=[61696]\n'
' bw ( KiB/s): min=106032, '
'max=106788, per=100.00%, '
'avg=106351.57, stdev=183.47, '
'samples=14\n'
' iops : min=212064, '
'max=213577, avg=212703.21, '
'stdev=367.12, samples=14\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=97.37%, 20=2.51%, '
'50=0.09%\n'
' lat (usec) : 100=0.02%\n'
' cpu : usr=15.79%, '
'sys=46.29%, ctx=747296, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1488872,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=104MiB/s (109MB/s), '
'104MiB/s-104MiB/s '
'(109MB/s-109MB/s), io=727MiB '
'(762MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_114.output'),
'state': {'begin': 1666782593.0277514,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8107247352600098,
'end': 1666782593.8384762,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_114.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_114.state')},
'cmd_115': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_115.output'),
'state': {'begin': 1666782594.901026,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7831172943115234,
'end': 1666782595.6841433,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_115.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_115.state')},
'cmd_116': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_116.output'),
'state': {'begin': 1666782595.6861565,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.10537338256836,
'end': 1666782606.79153,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_116.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_116.state')},
'cmd_117': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789696,\n'
' "timestamp_ms" : '
'1666789696879,\n'
' "time" : "Wed Oct 26 15:08:16 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 762843136,\n'
' "io_kbytes" : 744964,\n'
' "bw_bytes" : 108977590,\n'
' "bw" : 106423,\n'
' "iops" : 212846.714286,\n'
' "runtime" : 7000,\n'
' "total_ios" : 1489927,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1212,\n'
' "max" : 34444,\n'
' "mean" : 1402.238872,\n'
' "stddev" : '
'161.624220,\n'
' "N" : 1489927\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 180,\n'
' "max" : 110517,\n'
' "mean" : 7730.380643,\n'
' "stddev" : '
'2149.553208,\n'
' "N" : 1489928,\n'
' "percentile" : {\n'
' "1.000000" : 5536,\n'
' "5.000000" : 5664,\n'
' "10.000000" : 5728,\n'
' "20.000000" : 5792,\n'
' "30.000000" : 5920,\n'
' "40.000000" : 7264,\n'
' "50.000000" : 7520,\n'
' "60.000000" : 7712,\n'
' "70.000000" : 9280,\n'
' "80.000000" : 9408,\n'
' "90.000000" : 9408,\n'
' "95.000000" : 9536,\n'
' "99.000000" : '
'16768,\n'
' "99.500000" : '
'18304,\n'
' "99.900000" : '
'23424,\n'
' "99.950000" : '
'24192,\n'
' "99.990000" : 63744\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5690,\n'
' "max" : 111760,\n'
' "mean" : 9132.619582,\n'
' "stddev" : '
'2205.875773,\n'
' "N" : 1489928\n'
' },\n'
' "bw_min" : 106063,\n'
' "bw_max" : 106683,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'106427.071429,\n'
' "bw_dev" : 168.812807,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 212126,\n'
' "iops_max" : 213366,\n'
' "iops_mean" : '
'212854.142857,\n'
' "iops_stddev" : '
'337.625614,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.314286,\n'
' "sys_cpu" : 48.085714,\n'
' "ctx" : 747909,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.368864,\n'
' "20" : 2.503680,\n'
' "50" : 0.096649,\n'
' "100" : 0.021276,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=6880: Wed '
'Oct 26 15:08:16 2022\n'
' read: IOPS=213k, BW=104MiB/s '
'(109MB/s)(728MiB/7000msec)\n'
' slat (nsec): min=1212, '
'max=34444, avg=1402.24, '
'stdev=161.62\n'
' clat (nsec): min=180, '
'max=110517, avg=7730.38, '
'stdev=2149.55\n'
' lat (usec): min=5, max=111, '
'avg= 9.13, stdev= 2.21\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5536], '
'5.00th=[ 5664], 10.00th=[ 5728], '
'20.00th=[ 5792],\n'
' | 30.00th=[ 5920], '
'40.00th=[ 7264], 50.00th=[ '
'7520], 60.00th=[ 7712],\n'
' | 70.00th=[ 9280], '
'80.00th=[ 9408], 90.00th=[ '
'9408], 95.00th=[ 9536],\n'
' | 99.00th=[16768], '
'99.50th=[18304], '
'99.90th=[23424], '
'99.95th=[24192],\n'
' | 99.99th=[63744]\n'
' bw ( KiB/s): min=106063, '
'max=106683, per=100.00%, '
'avg=106427.07, stdev=168.81, '
'samples=14\n'
' iops : min=212126, '
'max=213366, avg=212854.14, '
'stdev=337.63, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=97.37%, 20=2.50%, 50=0.10%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=14.31%, '
'sys=48.09%, ctx=747909, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1489927,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=104MiB/s (109MB/s), '
'104MiB/s-104MiB/s '
'(109MB/s-109MB/s), io=728MiB '
'(763MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_117.output'),
'state': {'begin': 1666782606.7955425,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8198788166046143,
'end': 1666782607.6154213,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_117.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_117.state')},
'cmd_118': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_118.output'),
'state': {'begin': 1666782609.787193,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8587770462036133,
'end': 1666782610.64597,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_118.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_118.state')},
'cmd_119': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_119.output'),
'state': {'begin': 1666782610.6478796,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.552554607391357,
'end': 1666782622.2004342,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_119.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_119.state')},
'cmd_12': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789189,\n'
' "timestamp_ms" : '
'1666789189890,\n'
' "time" : "Wed Oct 26 14:59:49 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 394787328,\n'
' "io_kbytes" : 385534,\n'
' "bw_bytes" : 56390133,\n'
' "bw" : 55068,\n'
' "iops" : 110136.980431,\n'
' "runtime" : 7001,\n'
' "total_ios" : 771069,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1402,\n'
' "max" : 10500,\n'
' "mean" : 1496.382610,\n'
' "stddev" : 66.312543,\n'
' "N" : 771069\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 211,\n'
' "max" : 101751,\n'
' "mean" : 7283.762805,\n'
' "stddev" : '
'1020.270135,\n'
' "N" : 771069,\n'
' "percentile" : {\n'
' "1.000000" : 6752,\n'
' "5.000000" : 7008,\n'
' "10.000000" : 7072,\n'
' "20.000000" : 7072,\n'
' "30.000000" : 7072,\n'
' "40.000000" : 7136,\n'
' "50.000000" : 7200,\n'
' "60.000000" : 7200,\n'
' "70.000000" : 7264,\n'
' "80.000000" : 7264,\n'
' "90.000000" : 7392,\n'
' "95.000000" : 7456,\n'
' "99.000000" : 10560,\n'
' "99.500000" : 13504,\n'
' "99.900000" : 15680,\n'
' "99.950000" : 15936,\n'
' "99.990000" : 58624\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7324,\n'
' "max" : 103243,\n'
' "mean" : 8780.145415,\n'
' "stddev" : '
'1023.065350,\n'
' "N" : 771069\n'
' },\n'
' "bw_min" : 54847,\n'
' "bw_max" : 55341,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'55095.571429,\n'
' "bw_dev" : 155.343646,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 109694,\n'
' "iops_max" : 110683,\n'
' "iops_mean" : '
'110191.142857,\n'
' "iops_stddev" : '
'310.701404,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 10.771429,\n'
' "sys_cpu" : 33.328571,\n'
' "ctx" : 771067,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 98.642275,\n'
' "20" : 1.335289,\n'
' "50" : 0.010000,\n'
' "100" : 0.016211,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=5073: Wed '
'Oct 26 14:59:49 2022\n'
' read: IOPS=110k, BW=53.8MiB/s '
'(56.4MB/s)(376MiB/7001msec)\n'
' slat (nsec): min=1402, '
'max=10500, avg=1496.38, '
'stdev=66.31\n'
' clat (nsec): min=211, '
'max=101751, avg=7283.76, '
'stdev=1020.27\n'
' lat (usec): min=7, max=103, '
'avg= 8.78, stdev= 1.02\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6752], 5.00th=[ '
'7008], 10.00th=[ 7072], 20.00th=[ '
'7072],\n'
' | 30.00th=[ 7072], 40.00th=[ '
'7136], 50.00th=[ 7200], 60.00th=[ '
'7200],\n'
' | 70.00th=[ 7264], 80.00th=[ '
'7264], 90.00th=[ 7392], 95.00th=[ '
'7456],\n'
' | 99.00th=[10560], '
'99.50th=[13504], 99.90th=[15680], '
'99.95th=[15936],\n'
' | 99.99th=[58624]\n'
' bw ( KiB/s): min=54847, '
'max=55341, per=100.00%, '
'avg=55095.57, stdev=155.34, '
'samples=14\n'
' iops : min=109694, '
'max=110683, avg=110191.14, '
'stdev=310.70, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=98.64%, 20=1.34%, '
'50=0.01%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=10.77%, '
'sys=33.33%, ctx=771067, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=771069,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=53.8MiB/s (56.4MB/s), '
'53.8MiB/s-53.8MiB/s '
'(56.4MB/s-56.4MB/s), io=376MiB '
'(395MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_12.output'),
'state': {'begin': 1666782099.7968102,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8355357646942139,
'end': 1666782100.632346,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_12.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_12.state')},
'cmd_120': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789712,\n'
' "timestamp_ms" : '
'1666789712219,\n'
' "time" : "Wed Oct 26 15:08:32 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 909902848,\n'
' "io_kbytes" : 888577,\n'
' "bw_bytes" : 129967554,\n'
' "bw" : 126921,\n'
' "iops" : 253842.736752,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1777153,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 220,\n'
' "max" : 16330,\n'
' "mean" : 271.297074,\n'
' "stddev" : 24.250548,\n'
' "N" : 1777153\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 281,\n'
' "max" : 121247,\n'
' "mean" : 7381.103358,\n'
' "stddev" : '
'1757.802881,\n'
' "N" : 1777154,\n'
' "percentile" : {\n'
' "1.000000" : 5984,\n'
' "5.000000" : 6240,\n'
' "10.000000" : 6368,\n'
' "20.000000" : 6624,\n'
' "30.000000" : 7072,\n'
' "40.000000" : 7136,\n'
' "50.000000" : 7200,\n'
' "60.000000" : 7264,\n'
' "70.000000" : 7328,\n'
' "80.000000" : 7520,\n'
' "90.000000" : 8032,\n'
' "95.000000" : 8256,\n'
' "99.000000" : '
'16064,\n'
' "99.500000" : '
'20352,\n'
' "99.900000" : '
'22144,\n'
' "99.950000" : '
'22400,\n'
' "99.990000" : 63232\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5480,\n'
' "max" : 121528,\n'
' "mean" : 7652.400431,\n'
' "stddev" : '
'1758.138025,\n'
' "N" : 1777154\n'
' },\n'
' "bw_min" : 125453,\n'
' "bw_max" : 129526,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'126965.500000,\n'
' "bw_dev" : 1491.929507,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 250906,\n'
' "iops_max" : 259052,\n'
' "iops_mean" : '
'253931.000000,\n'
' "iops_stddev" : '
'2984.065734,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 17,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 99.985314,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.055459,\n'
' "20" : 2.406096,\n'
' "50" : 0.517007,\n'
' "100" : 0.020651,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=6940: Wed Oct 26 15:08:32 '
'2022\n'
' read: IOPS=254k, BW=124MiB/s '
'(130MB/s)(868MiB/7001msec)\n'
' slat (nsec): min=220, '
'max=16330, avg=271.30, '
'stdev=24.25\n'
' clat (nsec): min=281, '
'max=121247, avg=7381.10, '
'stdev=1757.80\n'
' lat (usec): min=5, max=121, '
'avg= 7.65, stdev= 1.76\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5984], '
'5.00th=[ 6240], 10.00th=[ 6368], '
'20.00th=[ 6624],\n'
' | 30.00th=[ 7072], '
'40.00th=[ 7136], 50.00th=[ '
'7200], 60.00th=[ 7264],\n'
' | 70.00th=[ 7328], '
'80.00th=[ 7520], 90.00th=[ '
'8032], 95.00th=[ 8256],\n'
' | 99.00th=[16064], '
'99.50th=[20352], '
'99.90th=[22144], '
'99.95th=[22400],\n'
' | 99.99th=[63232]\n'
' bw ( KiB/s): min=125453, '
'max=129526, per=100.00%, '
'avg=126965.50, stdev=1491.93, '
'samples=14\n'
' iops : min=250906, '
'max=259052, avg=253931.00, '
'stdev=2984.07, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=97.06%, 20=2.41%, 50=0.52%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=17, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1777153,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=124MiB/s (130MB/s), '
'124MiB/s-124MiB/s '
'(130MB/s-130MB/s), io=868MiB '
'(910MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_120.output'),
'state': {'begin': 1666782622.2029004,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8285534381866455,
'end': 1666782623.0314538,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_120.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_120.state')},
'cmd_121': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_121.output'),
'state': {'begin': 1666782625.0079858,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8558175563812256,
'end': 1666782625.8638034,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_121.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_121.state')},
'cmd_122': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_122.output'),
'state': {'begin': 1666782625.865948,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.472501516342163,
'end': 1666782637.3384495,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_122.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_122.state')},
'cmd_123': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789727,\n'
' "timestamp_ms" : '
'1666789727362,\n'
' "time" : "Wed Oct 26 15:08:47 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 699008512,\n'
' "io_kbytes" : 682625,\n'
' "bw_bytes" : 99844095,\n'
' "bw" : 97503,\n'
' "iops" : 195007.856021,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1365250,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 711,\n'
' "max" : 12062,\n'
' "mean" : 936.282706,\n'
' "stddev" : 69.667152,\n'
' "N" : 1365250\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5841,\n'
' "max" : 82906,\n'
' "mean" : 9006.613612,\n'
' "stddev" : '
'1552.777792,\n'
' "N" : 1365251,\n'
' "percentile" : {\n'
' "1.000000" : 7264,\n'
' "5.000000" : 8032,\n'
' "10.000000" : 8512,\n'
' "20.000000" : 8640,\n'
' "30.000000" : 8768,\n'
' "40.000000" : 8768,\n'
' "50.000000" : 8768,\n'
' "60.000000" : 8896,\n'
' "70.000000" : 9024,\n'
' "80.000000" : 9024,\n'
' "90.000000" : 9280,\n'
' "95.000000" : 9536,\n'
' "99.000000" : '
'17024,\n'
' "99.500000" : '
'18304,\n'
' "99.900000" : '
'24704,\n'
' "99.950000" : '
'25728,\n'
' "99.990000" : 62208\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7314,\n'
' "max" : 83837,\n'
' "mean" : 9942.896300,\n'
' "stddev" : '
'1554.645663,\n'
' "N" : 1365251\n'
' },\n'
' "bw_min" : 97305,\n'
' "bw_max" : 98105,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'97526.428571,\n'
' "bw_dev" : 203.481889,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 194610,\n'
' "iops_max" : 196210,\n'
' "iops_mean" : '
'195052.857143,\n'
' "iops_stddev" : '
'406.963778,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 16,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.534627,\n'
' "2" : 99.465373,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.686321,\n'
' "20" : 2.956968,\n'
' "50" : 0.336129,\n'
' "100" : 0.020656,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=6998: Wed Oct 26 15:08:47 '
'2022\n'
' read: IOPS=195k, BW=95.2MiB/s '
'(99.8MB/s)(667MiB/7001msec)\n'
' slat (nsec): min=711, '
'max=12062, avg=936.28, '
'stdev=69.67\n'
' clat (nsec): min=5841, '
'max=82906, avg=9006.61, '
'stdev=1552.78\n'
' lat (nsec): min=7314, '
'max=83837, avg=9942.90, '
'stdev=1554.65\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7264], '
'5.00th=[ 8032], 10.00th=[ 8512], '
'20.00th=[ 8640],\n'
' | 30.00th=[ 8768], '
'40.00th=[ 8768], 50.00th=[ '
'8768], 60.00th=[ 8896],\n'
' | 70.00th=[ 9024], '
'80.00th=[ 9024], 90.00th=[ '
'9280], 95.00th=[ 9536],\n'
' | 99.00th=[17024], '
'99.50th=[18304], '
'99.90th=[24704], '
'99.95th=[25728],\n'
' | 99.99th=[62208]\n'
' bw ( KiB/s): min=97305, '
'max=98105, per=100.00%, '
'avg=97526.43, stdev=203.48, '
'samples=14\n'
' iops : min=194610, '
'max=196210, avg=195052.86, '
'stdev=406.96, samples=14\n'
' lat (usec) : 10=96.69%, '
'20=2.96%, 50=0.34%, 100=0.02%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=16, majf=0, '
'minf=0\n'
' IO depths : 1=0.5%, '
'2=99.5%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1365250,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=95.2MiB/s '
'(99.8MB/s), 95.2MiB/s-95.2MiB/s '
'(99.8MB/s-99.8MB/s), io=667MiB '
'(699MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_123.output'),
'state': {'begin': 1666782637.3410268,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8542912006378174,
'end': 1666782638.195318,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_123.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_123.state')},
'cmd_124': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_124.output'),
'state': {'begin': 1666782640.2805362,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8470785617828369,
'end': 1666782641.1276147,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_124.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_124.state')},
'cmd_125': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_125.output'),
'state': {'begin': 1666782641.12973,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.52648115158081,
'end': 1666782652.6562111,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_125.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_125.state')},
'cmd_126': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789742,\n'
' "timestamp_ms" : '
'1666789742679,\n'
' "time" : "Wed Oct 26 15:09:02 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 808450048,\n'
' "io_kbytes" : 789502,\n'
' "bw_bytes" : 115476367,\n'
' "bw" : 112769,\n'
' "iops" : 225539.637195,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1579003,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 672,\n'
' "max" : 21169,\n'
' "mean" : 953.871964,\n'
' "stddev" : '
'122.852261,\n'
' "N" : 1579003\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 752,\n'
' "max" : 110307,\n'
' "mean" : 7658.542619,\n'
' "stddev" : '
'1855.562506,\n'
' "N" : 1579004,\n'
' "percentile" : {\n'
' "1.000000" : 6624,\n'
' "5.000000" : 6752,\n'
' "10.000000" : 6816,\n'
' "20.000000" : 6944,\n'
' "30.000000" : 7072,\n'
' "40.000000" : 7136,\n'
' "50.000000" : 7200,\n'
' "60.000000" : 7264,\n'
' "70.000000" : 7392,\n'
' "80.000000" : 7520,\n'
' "90.000000" : 9536,\n'
' "95.000000" : 9792,\n'
' "99.000000" : '
'17280,\n'
' "99.500000" : '
'19328,\n'
' "99.900000" : '
'20352,\n'
' "99.950000" : '
'22912,\n'
' "99.990000" : 62208\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7073,\n'
' "max" : 111219,\n'
' "mean" : 8612.414576,\n'
' "stddev" : '
'1850.718560,\n'
' "N" : 1579004\n'
' },\n'
' "bw_min" : 112339,\n'
' "bw_max" : 113330,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'112795.214286,\n'
' "bw_dev" : 372.083361,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 224678,\n'
' "iops_max" : 226660,\n'
' "iops_mean" : '
'225590.428571,\n'
' "iops_stddev" : '
'744.166722,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 12,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.296453,\n'
' "2" : 99.703547,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.579234,\n'
' "20" : 3.241919,\n'
' "50" : 0.156808,\n'
' "100" : 0.021849,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=7056: Wed Oct 26 15:09:02 '
'2022\n'
' read: IOPS=226k, BW=110MiB/s '
'(115MB/s)(771MiB/7001msec)\n'
' slat (nsec): min=672, '
'max=21169, avg=953.87, '
'stdev=122.85\n'
' clat (nsec): min=752, '
'max=110307, avg=7658.54, '
'stdev=1855.56\n'
' lat (usec): min=7, max=111, '
'avg= 8.61, stdev= 1.85\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6624], '
'5.00th=[ 6752], 10.00th=[ 6816], '
'20.00th=[ 6944],\n'
' | 30.00th=[ 7072], '
'40.00th=[ 7136], 50.00th=[ '
'7200], 60.00th=[ 7264],\n'
' | 70.00th=[ 7392], '
'80.00th=[ 7520], 90.00th=[ '
'9536], 95.00th=[ 9792],\n'
' | 99.00th=[17280], '
'99.50th=[19328], '
'99.90th=[20352], '
'99.95th=[22912],\n'
' | 99.99th=[62208]\n'
' bw ( KiB/s): min=112339, '
'max=113330, per=100.00%, '
'avg=112795.21, stdev=372.08, '
'samples=14\n'
' iops : min=224678, '
'max=226660, avg=225590.43, '
'stdev=744.17, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.58%, 20=3.24%, 50=0.16%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=12, majf=0, '
'minf=0\n'
' IO depths : 1=0.3%, '
'2=99.7%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1579003,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=110MiB/s (115MB/s), '
'110MiB/s-110MiB/s '
'(115MB/s-115MB/s), io=771MiB '
'(808MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_126.output'),
'state': {'begin': 1666782652.6588125,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8447775840759277,
'end': 1666782653.50359,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_126.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_126.state')},
'cmd_127': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_127.output'),
'state': {'begin': 1666782654.545266,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8357486724853516,
'end': 1666782655.3810146,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_127.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_127.state')},
'cmd_128': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_128.output'),
'state': {'begin': 1666782655.3829398,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.060334205627441,
'end': 1666782666.443274,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_128.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_128.state')},
'cmd_129': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789756,\n'
' "timestamp_ms" : '
'1666789756530,\n'
' "time" : "Wed Oct 26 15:09:16 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 763407360,\n'
' "io_kbytes" : 745515,\n'
' "bw_bytes" : 109058194,\n'
' "bw" : 106502,\n'
' "iops" : 213004.142857,\n'
' "runtime" : 7000,\n'
' "total_ios" : 1491029,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1312,\n'
' "max" : 15359,\n'
' "mean" : 1507.511643,\n'
' "stddev" : '
'157.070001,\n'
' "N" : 1491029\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1262,\n'
' "max" : 111940,\n'
' "mean" : 7604.344876,\n'
' "stddev" : '
'2256.544363,\n'
' "N" : 1491030,\n'
' "percentile" : {\n'
' "1.000000" : 5280,\n'
' "5.000000" : 5344,\n'
' "10.000000" : 5408,\n'
' "20.000000" : 5472,\n'
' "30.000000" : 5600,\n'
' "40.000000" : 7264,\n'
' "50.000000" : 7648,\n'
' "60.000000" : 8032,\n'
' "70.000000" : 9024,\n'
' "80.000000" : 9152,\n'
' "90.000000" : 9280,\n'
' "95.000000" : 9408,\n'
' "99.000000" : '
'16512,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'23936,\n'
' "99.950000" : '
'24192,\n'
' "99.990000" : 62720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5671,\n'
' "max" : 113352,\n'
' "mean" : 9111.856576,\n'
' "stddev" : '
'2313.110543,\n'
' "N" : 1491030\n'
' },\n'
' "bw_min" : 106106,\n'
' "bw_max" : 106959,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'106515.714286,\n'
' "bw_dev" : 259.918457,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 212212,\n'
' "iops_max" : 213918,\n'
' "iops_mean" : '
'213031.500000,\n'
' "iops_stddev" : '
'519.943007,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 12.800000,\n'
' "sys_cpu" : 50.100000,\n'
' "ctx" : 749016,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 97.365913,\n'
' "20" : 2.445761,\n'
' "50" : 0.154725,\n'
' "100" : 0.024144,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2140162,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 10972,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 10972,\n'
' "util" : 99.069123\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=7103: Wed '
'Oct 26 15:09:16 2022\n'
' read: IOPS=213k, BW=104MiB/s '
'(109MB/s)(728MiB/7000msec)\n'
' slat (nsec): min=1312, '
'max=15359, avg=1507.51, '
'stdev=157.07\n'
' clat (nsec): min=1262, '
'max=111940, avg=7604.34, '
'stdev=2256.54\n'
' lat (usec): min=5, max=113, '
'avg= 9.11, stdev= 2.31\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5280], '
'5.00th=[ 5344], 10.00th=[ 5408], '
'20.00th=[ 5472],\n'
' | 30.00th=[ 5600], '
'40.00th=[ 7264], 50.00th=[ '
'7648], 60.00th=[ 8032],\n'
' | 70.00th=[ 9024], '
'80.00th=[ 9152], 90.00th=[ '
'9280], 95.00th=[ 9408],\n'
' | 99.00th=[16512], '
'99.50th=[18560], '
'99.90th=[23936], '
'99.95th=[24192],\n'
' | 99.99th=[62720]\n'
' bw ( KiB/s): min=106106, '
'max=106959, per=100.00%, '
'avg=106515.71, stdev=259.92, '
'samples=14\n'
' iops : min=212212, '
'max=213918, avg=213031.50, '
'stdev=519.94, samples=14\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=97.37%, 20=2.45%, '
'50=0.15%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=12.80%, '
'sys=50.10%, ctx=749016, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1491029,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=104MiB/s (109MB/s), '
'104MiB/s-104MiB/s '
'(109MB/s-109MB/s), io=728MiB '
'(763MB), run=7000-7000msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2140162/0, '
'merge=0/0, ticks=10972/0, '
'in_queue=10972, util=99.07%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_129.output'),
'state': {'begin': 1666782666.445754,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8473761081695557,
'end': 1666782667.2931302,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_129.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_129.state')},
'cmd_13': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_13.output'),
'state': {'begin': 1666782101.6819694,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8460066318511963,
'end': 1666782102.527976,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_13.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_13.state')},
'cmd_130': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_130.output'),
'state': {'begin': 1666782668.350055,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8273735046386719,
'end': 1666782669.1774285,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_130.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_130.state')},
'cmd_131': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_131.output'),
'state': {'begin': 1666782669.179487,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.059387683868408,
'end': 1666782680.2388747,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_131.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_131.state')},
'cmd_132': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789770,\n'
' "timestamp_ms" : '
'1666789770326,\n'
' "time" : "Wed Oct 26 15:09:30 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 757835776,\n'
' "io_kbytes" : 740074,\n'
' "bw_bytes" : 108246789,\n'
' "bw" : 105709,\n'
' "iops" : 211419.368662,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1480147,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1332,\n'
' "max" : 26660,\n'
' "mean" : 1516.955765,\n'
' "stddev" : '
'165.330542,\n'
' "N" : 1480147\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 200,\n'
' "max" : 109896,\n'
' "mean" : 7661.495982,\n'
' "stddev" : '
'2256.062018,\n'
' "N" : 1480148,\n'
' "percentile" : {\n'
' "1.000000" : 5280,\n'
' "5.000000" : 5408,\n'
' "10.000000" : 5472,\n'
' "20.000000" : 5536,\n'
' "30.000000" : 5792,\n'
' "40.000000" : 7328,\n'
' "50.000000" : 7648,\n'
' "60.000000" : 7968,\n'
' "70.000000" : 9152,\n'
' "80.000000" : 9152,\n'
' "90.000000" : 9280,\n'
' "95.000000" : 9536,\n'
' "99.000000" : '
'16768,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'23936,\n'
' "99.950000" : '
'24448,\n'
' "99.990000" : 65280\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5771,\n'
' "max" : 111358,\n'
' "mean" : 9178.451778,\n'
' "stddev" : '
'2304.102186,\n'
' "N" : 1480148\n'
' },\n'
' "bw_min" : 105505,\n'
' "bw_max" : 106062,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'105778.000000,\n'
' "bw_dev" : 128.192764,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 211010,\n'
' "iops_max" : 212124,\n'
' "iops_mean" : '
'211556.142857,\n'
' "iops_stddev" : '
'256.392385,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.585714,\n'
' "sys_cpu" : 49.757143,\n'
' "ctx" : 743789,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.308105,\n'
' "20" : 2.463809,\n'
' "50" : 0.192211,\n'
' "100" : 0.026146,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2124569,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 10947,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 10947,\n'
' "util" : 99.068754\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=7150: Wed '
'Oct 26 15:09:30 2022\n'
' read: IOPS=211k, BW=103MiB/s '
'(108MB/s)(723MiB/7001msec)\n'
' slat (nsec): min=1332, '
'max=26660, avg=1516.96, '
'stdev=165.33\n'
' clat (nsec): min=200, '
'max=109896, avg=7661.50, '
'stdev=2256.06\n'
' lat (usec): min=5, max=111, '
'avg= 9.18, stdev= 2.30\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5280], '
'5.00th=[ 5408], 10.00th=[ 5472], '
'20.00th=[ 5536],\n'
' | 30.00th=[ 5792], '
'40.00th=[ 7328], 50.00th=[ '
'7648], 60.00th=[ 7968],\n'
' | 70.00th=[ 9152], '
'80.00th=[ 9152], 90.00th=[ '
'9280], 95.00th=[ 9536],\n'
' | 99.00th=[16768], '
'99.50th=[18560], '
'99.90th=[23936], '
'99.95th=[24448],\n'
' | 99.99th=[65280]\n'
' bw ( KiB/s): min=105505, '
'max=106062, per=100.00%, '
'avg=105778.00, stdev=128.19, '
'samples=14\n'
' iops : min=211010, '
'max=212124, avg=211556.14, '
'stdev=256.39, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=97.31%, 20=2.46%, 50=0.19%, '
'100=0.03%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=13.59%, '
'sys=49.76%, ctx=743789, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1480147,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=103MiB/s (108MB/s), '
'103MiB/s-103MiB/s '
'(108MB/s-108MB/s), io=723MiB '
'(758MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2124569/0, '
'merge=0/0, ticks=10947/0, '
'in_queue=10947, util=99.07%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_132.output'),
'state': {'begin': 1666782680.2413769,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8027191162109375,
'end': 1666782681.044096,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_132.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_132.state')},
'cmd_133': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_133.output'),
'state': {'begin': 1666782682.1035376,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.838510274887085,
'end': 1666782682.9420478,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_133.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_133.state')},
'cmd_134': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_134.output'),
'state': {'begin': 1666782682.9488225,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.067566871643066,
'end': 1666782694.0163894,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_134.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_134.state')},
'cmd_135': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789784,\n'
' "timestamp_ms" : '
'1666789784103,\n'
' "time" : "Wed Oct 26 15:09:44 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 760478720,\n'
' "io_kbytes" : 742655,\n'
' "bw_bytes" : 108624299,\n'
' "bw" : 106078,\n'
' "iops" : 212156.691901,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1485309,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1332,\n'
' "max" : 22673,\n'
' "mean" : 1514.181856,\n'
' "stddev" : '
'153.990339,\n'
' "N" : 1485309\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 100,\n'
' "max" : 117761,\n'
' "mean" : 7616.194574,\n'
' "stddev" : '
'2211.320112,\n'
' "N" : 1485310,\n'
' "percentile" : {\n'
' "1.000000" : 5280,\n'
' "5.000000" : 5344,\n'
' "10.000000" : 5344,\n'
' "20.000000" : 5472,\n'
' "30.000000" : 5664,\n'
' "40.000000" : 7328,\n'
' "50.000000" : 7648,\n'
' "60.000000" : 7968,\n'
' "70.000000" : 9024,\n'
' "80.000000" : 9152,\n'
' "90.000000" : 9280,\n'
' "95.000000" : 9408,\n'
' "99.000000" : '
'16512,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'23680,\n'
' "99.950000" : '
'24192,\n'
' "99.990000" : 59648\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5700,\n'
' "max" : 119204,\n'
' "mean" : 9130.376490,\n'
' "stddev" : '
'2257.397056,\n'
' "N" : 1485310\n'
' },\n'
' "bw_min" : 105355,\n'
' "bw_max" : 106441,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'106110.214286,\n'
' "bw_dev" : 316.062598,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 210711,\n'
' "iops_max" : 212882,\n'
' "iops_mean" : '
'212220.571429,\n'
' "iops_stddev" : '
'631.966980,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.228571,\n'
' "sys_cpu" : 50.228571,\n'
' "ctx" : 746364,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.327155,\n'
' "20" : 2.466490,\n'
' "50" : 0.177135,\n'
' "100" : 0.019592,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2126963,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 10923,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 10923,\n'
' "util" : 99.078934\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=7196: Wed '
'Oct 26 15:09:44 2022\n'
' read: IOPS=212k, BW=104MiB/s '
'(109MB/s)(725MiB/7001msec)\n'
' slat (nsec): min=1332, '
'max=22673, avg=1514.18, '
'stdev=153.99\n'
' clat (nsec): min=100, '
'max=117761, avg=7616.19, '
'stdev=2211.32\n'
' lat (usec): min=5, max=119, '
'avg= 9.13, stdev= 2.26\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5280], '
'5.00th=[ 5344], 10.00th=[ 5344], '
'20.00th=[ 5472],\n'
' | 30.00th=[ 5664], '
'40.00th=[ 7328], 50.00th=[ '
'7648], 60.00th=[ 7968],\n'
' | 70.00th=[ 9024], '
'80.00th=[ 9152], 90.00th=[ '
'9280], 95.00th=[ 9408],\n'
' | 99.00th=[16512], '
'99.50th=[18560], '
'99.90th=[23680], '
'99.95th=[24192],\n'
' | 99.99th=[59648]\n'
' bw ( KiB/s): min=105355, '
'max=106441, per=100.00%, '
'avg=106110.21, stdev=316.06, '
'samples=14\n'
' iops : min=210711, '
'max=212882, avg=212220.57, '
'stdev=631.97, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=97.33%, 20=2.47%, 50=0.18%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=13.23%, '
'sys=50.23%, ctx=746364, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1485309,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=104MiB/s (109MB/s), '
'104MiB/s-104MiB/s '
'(109MB/s-109MB/s), io=725MiB '
'(760MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2126963/0, '
'merge=0/0, ticks=10923/0, '
'in_queue=10923, util=99.08%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_135.output'),
'state': {'begin': 1666782694.018831,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.8572797775268555,
'end': 1666782694.8761108,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_135.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_135.state')},
'cmd_136': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_136.output'),
'state': {'begin': 1666782696.9613917,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.7837743759155273,
'end': 1666782697.745166,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_136.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_136.state')},
'cmd_137': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_137.output'),
'state': {'begin': 1666782697.7469563,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.493335962295532,
'end': 1666782709.2402923,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_137.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_137.state')},
'cmd_138': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789799,\n'
' "timestamp_ms" : '
'1666789799256,\n'
' "time" : "Wed Oct 26 15:09:59 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 928209408,\n'
' "io_kbytes" : 906454,\n'
' "bw_bytes" : 132582403,\n'
' "bw" : 129475,\n'
' "iops" : 258949.864305,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1812908,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 12924,\n'
' "mean" : 173.489903,\n'
' "stddev" : 32.365104,\n'
' "N" : 1812908\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5941,\n'
' "max" : 118202,\n'
' "mean" : 7276.558786,\n'
' "stddev" : '
'1678.838912,\n'
' "N" : 1812909,\n'
' "percentile" : {\n'
' "1.000000" : 6560,\n'
' "5.000000" : 6688,\n'
' "10.000000" : 6688,\n'
' "20.000000" : 6752,\n'
' "30.000000" : 6752,\n'
' "40.000000" : 6816,\n'
' "50.000000" : 6880,\n'
' "60.000000" : 6880,\n'
' "70.000000" : 6944,\n'
' "80.000000" : 7200,\n'
' "90.000000" : 8256,\n'
' "95.000000" : 8384,\n'
' "99.000000" : '
'14656,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'20352,\n'
' "99.950000" : '
'21120,\n'
' "99.990000" : 62720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6111,\n'
' "max" : 118372,\n'
' "mean" : 7450.048687,\n'
' "stddev" : '
'1675.358061,\n'
' "N" : 1812909\n'
' },\n'
' "bw_min" : 128981,\n'
' "bw_max" : 130259,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'129499.928571,\n'
' "bw_dev" : 474.433989,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 257963,\n'
' "iops_max" : 260518,\n'
' "iops_mean" : '
'258999.928571,\n'
' "iops_stddev" : '
'948.783875,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 32.957143,\n'
' "sys_cpu" : 67.014286,\n'
' "ctx" : 11,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 8.368213,\n'
' "2" : 91.631787,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 97.055173,\n'
' "20" : 2.794626,\n'
' "50" : 0.126316,\n'
' "100" : 0.023608,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=7252: Wed '
'Oct 26 15:09:59 2022\n'
' read: IOPS=259k, BW=126MiB/s '
'(133MB/s)(885MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=12924, avg=173.49, '
'stdev=32.37\n'
' clat (usec): min=5, max=118, '
'avg= 7.28, stdev= 1.68\n'
' lat (usec): min=6, max=118, '
'avg= 7.45, stdev= 1.68\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6560], '
'5.00th=[ 6688], 10.00th=[ 6688], '
'20.00th=[ 6752],\n'
' | 30.00th=[ 6752], '
'40.00th=[ 6816], 50.00th=[ '
'6880], 60.00th=[ 6880],\n'
' | 70.00th=[ 6944], '
'80.00th=[ 7200], 90.00th=[ '
'8256], 95.00th=[ 8384],\n'
' | 99.00th=[14656], '
'99.50th=[18560], '
'99.90th=[20352], '
'99.95th=[21120],\n'
' | 99.99th=[62720]\n'
' bw ( KiB/s): min=128981, '
'max=130259, per=100.00%, '
'avg=129499.93, stdev=474.43, '
'samples=14\n'
' iops : min=257963, '
'max=260518, avg=258999.93, '
'stdev=948.78, samples=14\n'
' lat (usec) : 10=97.06%, '
'20=2.79%, 50=0.13%, 100=0.02%, '
'250=0.01%\n'
' cpu : usr=32.96%, '
'sys=67.01%, ctx=11, majf=0, '
'minf=0\n'
' IO depths : 1=8.4%, '
'2=91.6%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1812908,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=126MiB/s (133MB/s), '
'126MiB/s-126MiB/s '
'(133MB/s-133MB/s), io=885MiB '
'(928MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_138.output'),
'state': {'begin': 1666782709.2428133,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.8369588851928711,
'end': 1666782710.0797722,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_138.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_138.state')},
'cmd_139': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_139.output'),
'state': {'begin': 1666782712.184676,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.8070030212402344,
'end': 1666782712.991679,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_139.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_139.state')},
'cmd_14': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_14.output'),
'state': {'begin': 1666782102.530145,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.075329542160034,
'end': 1666782113.6054745,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_14.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_14.state')},
'cmd_140': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_140.output'),
'state': {'begin': 1666782712.9987884,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.543930292129517,
'end': 1666782724.5427186,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_140.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_140.state')},
'cmd_141': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789814,\n'
' "timestamp_ms" : '
'1666789814565,\n'
' "time" : "Wed Oct 26 15:10:14 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 919181824,\n'
' "io_kbytes" : 897638,\n'
' "bw_bytes" : 131292933,\n'
' "bw" : 128215,\n'
' "iops" : 256431.366948,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1795276,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 11972,\n'
' "mean" : 171.000074,\n'
' "stddev" : 44.899254,\n'
' "N" : 1795276\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5971,\n'
' "max" : 119064,\n'
' "mean" : 7345.391732,\n'
' "stddev" : '
'1661.364011,\n'
' "N" : 1795277,\n'
' "percentile" : {\n'
' "1.000000" : 6624,\n'
' "5.000000" : 6688,\n'
' "10.000000" : 6752,\n'
' "20.000000" : 6816,\n'
' "30.000000" : 6816,\n'
' "40.000000" : 6880,\n'
' "50.000000" : 6880,\n'
' "60.000000" : 6944,\n'
' "70.000000" : 7008,\n'
' "80.000000" : 7840,\n'
' "90.000000" : 8256,\n'
' "95.000000" : 8512,\n'
' "99.000000" : '
'14784,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'20096,\n'
' "99.950000" : '
'20864,\n'
' "99.990000" : 62720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6151,\n'
' "max" : 119244,\n'
' "mean" : 7516.391811,\n'
' "stddev" : '
'1658.455431,\n'
' "N" : 1795277\n'
' },\n'
' "bw_min" : 127602,\n'
' "bw_max" : 128753,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'128239.071429,\n'
' "bw_dev" : 367.154225,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 255204,\n'
' "iops_max" : 257506,\n'
' "iops_mean" : '
'256478.214286,\n'
' "iops_stddev" : '
'734.226771,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 33.300000,\n'
' "sys_cpu" : 66.671429,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 9.490184,\n'
' "2" : 90.509816,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 97.017896,\n'
' "20" : 2.866412,\n'
' "50" : 0.094359,\n'
' "100" : 0.021055,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=7309: Wed '
'Oct 26 15:10:14 2022\n'
' read: IOPS=256k, BW=125MiB/s '
'(131MB/s)(877MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=11972, avg=171.00, '
'stdev=44.90\n'
' clat (usec): min=5, max=119, '
'avg= 7.35, stdev= 1.66\n'
' lat (usec): min=6, max=119, '
'avg= 7.52, stdev= 1.66\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6624], '
'5.00th=[ 6688], 10.00th=[ 6752], '
'20.00th=[ 6816],\n'
' | 30.00th=[ 6816], '
'40.00th=[ 6880], 50.00th=[ '
'6880], 60.00th=[ 6944],\n'
' | 70.00th=[ 7008], '
'80.00th=[ 7840], 90.00th=[ '
'8256], 95.00th=[ 8512],\n'
' | 99.00th=[14784], '
'99.50th=[18560], '
'99.90th=[20096], '
'99.95th=[20864],\n'
' | 99.99th=[62720]\n'
' bw ( KiB/s): min=127602, '
'max=128753, per=100.00%, '
'avg=128239.07, stdev=367.15, '
'samples=14\n'
' iops : min=255204, '
'max=257506, avg=256478.21, '
'stdev=734.23, samples=14\n'
' lat (usec) : 10=97.02%, '
'20=2.87%, 50=0.09%, 100=0.02%, '
'250=0.01%\n'
' cpu : usr=33.30%, '
'sys=66.67%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=9.5%, '
'2=90.5%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1795276,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=125MiB/s (131MB/s), '
'125MiB/s-125MiB/s '
'(131MB/s-131MB/s), io=877MiB '
'(919MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_141.output'),
'state': {'begin': 1666782724.5457072,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.8194141387939453,
'end': 1666782725.3651214,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_141.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_141.state')},
'cmd_142': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_142.output'),
'state': {'begin': 1666782727.4584682,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.7973694801330566,
'end': 1666782728.2558377,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_142.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_142.state')},
'cmd_143': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_143.output'),
'state': {'begin': 1666782728.2577453,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.487537860870361,
'end': 1666782739.7452831,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_143.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_143.state')},
'cmd_144': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789829,\n'
' "timestamp_ms" : '
'1666789829766,\n'
' "time" : "Wed Oct 26 15:10:29 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 924888576,\n'
' "io_kbytes" : 903211,\n'
' "bw_bytes" : 132108066,\n'
' "bw" : 129011,\n'
' "iops" : 258023.425225,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1806422,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 11772,\n'
' "mean" : 169.994528,\n'
' "stddev" : 41.933383,\n'
' "N" : 1806422\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5952,\n'
' "max" : 102713,\n'
' "mean" : 7298.990791,\n'
' "stddev" : '
'1647.972135,\n'
' "N" : 1806422,\n'
' "percentile" : {\n'
' "1.000000" : 6624,\n'
' "5.000000" : 6688,\n'
' "10.000000" : 6752,\n'
' "20.000000" : 6752,\n'
' "30.000000" : 6816,\n'
' "40.000000" : 6816,\n'
' "50.000000" : 6880,\n'
' "60.000000" : 6880,\n'
' "70.000000" : 6944,\n'
' "80.000000" : 7264,\n'
' "90.000000" : 8256,\n'
' "95.000000" : 8384,\n'
' "99.000000" : '
'14784,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'20352,\n'
' "99.950000" : '
'20864,\n'
' "99.990000" : 62208\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6122,\n'
' "max" : 102883,\n'
' "mean" : 7468.985319,\n'
' "stddev" : '
'1644.499905,\n'
' "N" : 1806422\n'
' },\n'
' "bw_min" : 125789,\n'
' "bw_max" : 129837,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'129033.714286,\n'
' "bw_dev" : 1030.428994,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 251579,\n'
' "iops_max" : 259674,\n'
' "iops_mean" : '
'258067.500000,\n'
' "iops_stddev" : '
'2060.615769,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 36.842857,\n'
' "sys_cpu" : 63.128571,\n'
' "ctx" : 16,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 8.782610,\n'
' "2" : 91.217390,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 97.043991,\n'
' "20" : 2.821212,\n'
' "50" : 0.114259,\n'
' "100" : 0.020427,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=7367: Wed '
'Oct 26 15:10:29 2022\n'
' read: IOPS=258k, BW=126MiB/s '
'(132MB/s)(882MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=11772, avg=169.99, '
'stdev=41.93\n'
' clat (usec): min=5, max=102, '
'avg= 7.30, stdev= 1.65\n'
' lat (usec): min=6, max=102, '
'avg= 7.47, stdev= 1.64\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6624], '
'5.00th=[ 6688], 10.00th=[ 6752], '
'20.00th=[ 6752],\n'
' | 30.00th=[ 6816], '
'40.00th=[ 6816], 50.00th=[ '
'6880], 60.00th=[ 6880],\n'
' | 70.00th=[ 6944], '
'80.00th=[ 7264], 90.00th=[ '
'8256], 95.00th=[ 8384],\n'
' | 99.00th=[14784], '
'99.50th=[18560], '
'99.90th=[20352], '
'99.95th=[20864],\n'
' | 99.99th=[62208]\n'
' bw ( KiB/s): min=125789, '
'max=129837, per=100.00%, '
'avg=129033.71, stdev=1030.43, '
'samples=14\n'
' iops : min=251579, '
'max=259674, avg=258067.50, '
'stdev=2060.62, samples=14\n'
' lat (usec) : 10=97.04%, '
'20=2.82%, 50=0.11%, 100=0.02%, '
'250=0.01%\n'
' cpu : usr=36.84%, '
'sys=63.13%, ctx=16, majf=0, '
'minf=0\n'
' IO depths : 1=8.8%, '
'2=91.2%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1806422,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=126MiB/s (132MB/s), '
'126MiB/s-126MiB/s '
'(132MB/s-132MB/s), io=882MiB '
'(925MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_144.output'),
'state': {'begin': 1666782739.7478065,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.8531312942504883,
'end': 1666782740.6009378,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_144.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_144.state')},
'cmd_145': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_145.output'),
'state': {'begin': 1666782742.8703396,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8204746246337891,
'end': 1666782743.6908143,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_145.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_145.state')},
'cmd_146': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_146.output'),
'state': {'begin': 1666782743.6930044,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.546950101852417,
'end': 1666782755.2399545,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_146.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_146.state')},
'cmd_147': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789845,\n'
' "timestamp_ms" : '
'1666789845262,\n'
' "time" : "Wed Oct 26 15:10:45 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1071219200,\n'
' "io_kbytes" : 1046112,\n'
' "bw_bytes" : 153009455,\n'
' "bw" : 149423,\n'
' "iops" : 298846.307670,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2092223,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1683,\n'
' "max" : 35506,\n'
' "mean" : 1886.948066,\n'
' "stddev" : '
'122.911174,\n'
' "N" : 2092224\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 260,\n'
' "max" : 119775,\n'
' "mean" : 4572.451873,\n'
' "stddev" : '
'1788.671868,\n'
' "N" : 2092224,\n'
' "percentile" : {\n'
' "1.000000" : 3888,\n'
' "5.000000" : 4128,\n'
' "10.000000" : 4192,\n'
' "20.000000" : 4256,\n'
' "30.000000" : 4256,\n'
' "40.000000" : 4320,\n'
' "50.000000" : 4320,\n'
' "60.000000" : 4320,\n'
' "70.000000" : 4384,\n'
' "80.000000" : 4448,\n'
' "90.000000" : 4448,\n'
' "95.000000" : 4640,\n'
' "99.000000" : '
'12608,\n'
' "99.500000" : '
'17536,\n'
' "99.900000" : '
'21888,\n'
' "99.950000" : '
'22144,\n'
' "99.990000" : 63232\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5139,\n'
' "max" : 121608,\n'
' "mean" : 6459.399939,\n'
' "stddev" : '
'1792.133017,\n'
' "N" : 2092224\n'
' },\n'
' "bw_min" : 148978,\n'
' "bw_max" : 149936,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'149450.785714,\n'
' "bw_dev" : 305.810695,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 297956,\n'
' "iops_max" : 299873,\n'
' "iops_mean" : '
'298901.642857,\n'
' "iops_stddev" : '
'612.123116,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 44.157143,\n'
' "sys_cpu" : 55.828571,\n'
' "ctx" : 13,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 3.109707,\n'
' "10" : 94.906518,\n'
' "20" : 1.720562,\n'
' "50" : 0.241657,\n'
' "100" : 0.020982,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=7424: Wed '
'Oct 26 15:10:45 2022\n'
' read: IOPS=299k, BW=146MiB/s '
'(153MB/s)(1022MiB/7001msec)\n'
' slat (nsec): min=1683, '
'max=35506, avg=1886.95, '
'stdev=122.91\n'
' clat (nsec): min=260, '
'max=119775, avg=4572.45, '
'stdev=1788.67\n'
' lat (usec): min=5, max=121, '
'avg= 6.46, stdev= 1.79\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3888], '
'5.00th=[ 4128], 10.00th=[ 4192], '
'20.00th=[ 4256],\n'
' | 30.00th=[ 4256], '
'40.00th=[ 4320], 50.00th=[ '
'4320], 60.00th=[ 4320],\n'
' | 70.00th=[ 4384], '
'80.00th=[ 4448], 90.00th=[ '
'4448], 95.00th=[ 4640],\n'
' | 99.00th=[12608], '
'99.50th=[17536], '
'99.90th=[21888], '
'99.95th=[22144],\n'
' | 99.99th=[63232]\n'
' bw ( KiB/s): min=148978, '
'max=149936, per=100.00%, '
'avg=149450.79, stdev=305.81, '
'samples=14\n'
' iops : min=297956, '
'max=299873, avg=298901.64, '
'stdev=612.12, samples=14\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%\n'
' lat (usec) : 4=3.11%, '
'10=94.91%, 20=1.72%, 50=0.24%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=44.16%, '
'sys=55.83%, ctx=13, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2092223,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=146MiB/s (153MB/s), '
'146MiB/s-146MiB/s '
'(153MB/s-153MB/s), io=1022MiB '
'(1071MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_147.output'),
'state': {'begin': 1666782755.242614,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8503968715667725,
'end': 1666782756.093011,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_147.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_147.state')},
'cmd_148': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_148.output'),
'state': {'begin': 1666782758.268324,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8301882743835449,
'end': 1666782759.0985122,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_148.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_148.state')},
'cmd_149': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_149.output'),
'state': {'begin': 1666782759.1007652,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.452395677566528,
'end': 1666782770.553161,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_149.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_149.state')},
'cmd_15': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789203,\n'
' "timestamp_ms" : '
'1666789203700,\n'
' "time" : "Wed Oct 26 15:00:03 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 394408960,\n'
' "io_kbytes" : 385165,\n'
' "bw_bytes" : 56336089,\n'
' "bw" : 55015,\n'
' "iops" : 110031.424082,\n'
' "runtime" : 7001,\n'
' "total_ios" : 770330,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1402,\n'
' "max" : 14557,\n'
' "mean" : 1512.964761,\n'
' "stddev" : 62.706972,\n'
' "N" : 770330\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 260,\n'
' "max" : 97473,\n'
' "mean" : 7280.894260,\n'
' "stddev" : 982.896349,\n'
' "N" : 770330,\n'
' "percentile" : {\n'
' "1.000000" : 6752,\n'
' "5.000000" : 7008,\n'
' "10.000000" : 7072,\n'
' "20.000000" : 7072,\n'
' "30.000000" : 7072,\n'
' "40.000000" : 7136,\n'
' "50.000000" : 7200,\n'
' "60.000000" : 7200,\n'
' "70.000000" : 7264,\n'
' "80.000000" : 7264,\n'
' "90.000000" : 7392,\n'
' "95.000000" : 7456,\n'
' "99.000000" : 10432,\n'
' "99.500000" : 13632,\n'
' "99.900000" : 15680,\n'
' "99.950000" : 15808,\n'
' "99.990000" : 58112\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6993,\n'
' "max" : 98975,\n'
' "mean" : 8793.859020,\n'
' "stddev" : 985.772717,\n'
' "N" : 770330\n'
' },\n'
' "bw_min" : 54788,\n'
' "bw_max" : 55269,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'55025.500000,\n'
' "bw_dev" : 130.894408,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 109576,\n'
' "iops_max" : 110538,\n'
' "iops_mean" : '
'110051.000000,\n'
' "iops_stddev" : '
'261.788817,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 8.414286,\n'
' "sys_cpu" : 35.785714,\n'
' "ctx" : 770332,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 98.690691,\n'
' "20" : 1.290486,\n'
' "50" : 0.010000,\n'
' "100" : 0.014799,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=5122: Wed '
'Oct 26 15:00:03 2022\n'
' read: IOPS=110k, BW=53.7MiB/s '
'(56.3MB/s)(376MiB/7001msec)\n'
' slat (nsec): min=1402, '
'max=14557, avg=1512.96, '
'stdev=62.71\n'
' clat (nsec): min=260, '
'max=97473, avg=7280.89, '
'stdev=982.90\n'
' lat (nsec): min=6993, '
'max=98975, avg=8793.86, '
'stdev=985.77\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6752], 5.00th=[ '
'7008], 10.00th=[ 7072], 20.00th=[ '
'7072],\n'
' | 30.00th=[ 7072], 40.00th=[ '
'7136], 50.00th=[ 7200], 60.00th=[ '
'7200],\n'
' | 70.00th=[ 7264], 80.00th=[ '
'7264], 90.00th=[ 7392], 95.00th=[ '
'7456],\n'
' | 99.00th=[10432], '
'99.50th=[13632], 99.90th=[15680], '
'99.95th=[15808],\n'
' | 99.99th=[58112]\n'
' bw ( KiB/s): min=54788, '
'max=55269, per=100.00%, '
'avg=55025.50, stdev=130.89, '
'samples=14\n'
' iops : min=109576, '
'max=110538, avg=110051.00, '
'stdev=261.79, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=98.69%, 20=1.29%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%\n'
' cpu : usr=8.41%, '
'sys=35.79%, ctx=770332, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=770330,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=53.7MiB/s (56.3MB/s), '
'53.7MiB/s-53.7MiB/s '
'(56.3MB/s-56.3MB/s), io=376MiB '
'(394MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_15.output'),
'state': {'begin': 1666782113.6086535,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8623161315917969,
'end': 1666782114.4709697,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_15.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_15.state')},
'cmd_150': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789860,\n'
' "timestamp_ms" : '
'1666789860552,\n'
' "time" : "Wed Oct 26 15:11:00 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1074551808,\n'
' "io_kbytes" : 1049367,\n'
' "bw_bytes" : 153485474,\n'
' "bw" : 149888,\n'
' "iops" : 299776.174832,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2098733,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1673,\n'
' "max" : 23223,\n'
' "mean" : 1867.469730,\n'
' "stddev" : '
'122.869433,\n'
' "N" : 2098733\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 230,\n'
' "max" : 115426,\n'
' "mean" : 4579.960121,\n'
' "stddev" : '
'1762.455468,\n'
' "N" : 2098734,\n'
' "percentile" : {\n'
' "1.000000" : 3888,\n'
' "5.000000" : 4128,\n'
' "10.000000" : 4192,\n'
' "20.000000" : 4256,\n'
' "30.000000" : 4256,\n'
' "40.000000" : 4320,\n'
' "50.000000" : 4320,\n'
' "60.000000" : 4384,\n'
' "70.000000" : 4384,\n'
' "80.000000" : 4448,\n'
' "90.000000" : 4512,\n'
' "95.000000" : 4640,\n'
' "99.000000" : '
'12608,\n'
' "99.500000" : '
'17536,\n'
' "99.900000" : '
'21888,\n'
' "99.950000" : '
'22400,\n'
' "99.990000" : 61184\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5070,\n'
' "max" : 117269,\n'
' "mean" : 6447.429815,\n'
' "stddev" : '
'1765.832177,\n'
' "N" : 2098734\n'
' },\n'
' "bw_min" : 149368,\n'
' "bw_max" : 150878,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'149916.857143,\n'
' "bw_dev" : 485.029080,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 298736,\n'
' "iops_max" : 301757,\n'
' "iops_mean" : '
'299833.785714,\n'
' "iops_stddev" : '
'970.210618,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 28.900000,\n'
' "sys_cpu" : 71.085714,\n'
' "ctx" : 9,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 3.019584,\n'
' "10" : 94.994837,\n'
' "20" : 1.723421,\n'
' "50" : 0.241717,\n'
' "100" : 0.019631,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=7482: Wed '
'Oct 26 15:11:00 2022\n'
' read: IOPS=300k, BW=146MiB/s '
'(153MB/s)(1025MiB/7001msec)\n'
' slat (nsec): min=1673, '
'max=23223, avg=1867.47, '
'stdev=122.87\n'
' clat (nsec): min=230, '
'max=115426, avg=4579.96, '
'stdev=1762.46\n'
' lat (usec): min=5, max=117, '
'avg= 6.45, stdev= 1.77\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3888], '
'5.00th=[ 4128], 10.00th=[ 4192], '
'20.00th=[ 4256],\n'
' | 30.00th=[ 4256], '
'40.00th=[ 4320], 50.00th=[ '
'4320], 60.00th=[ 4384],\n'
' | 70.00th=[ 4384], '
'80.00th=[ 4448], 90.00th=[ '
'4512], 95.00th=[ 4640],\n'
' | 99.00th=[12608], '
'99.50th=[17536], '
'99.90th=[21888], '
'99.95th=[22400],\n'
' | 99.99th=[61184]\n'
' bw ( KiB/s): min=149368, '
'max=150878, per=100.00%, '
'avg=149916.86, stdev=485.03, '
'samples=14\n'
' iops : min=298736, '
'max=301757, avg=299833.79, '
'stdev=970.21, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%, 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=3.02%, 10=94.99%, 20=1.72%, '
'50=0.24%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=28.90%, '
'sys=71.09%, ctx=9, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2098733,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=146MiB/s (153MB/s), '
'146MiB/s-146MiB/s '
'(153MB/s-153MB/s), io=1025MiB '
'(1075MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_150.output'),
'state': {'begin': 1666782770.5560105,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.801260232925415,
'end': 1666782771.3572707,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_150.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_150.state')},
'cmd_151': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_151.output'),
'state': {'begin': 1666782773.4524174,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7857146263122559,
'end': 1666782774.238132,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_151.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_151.state')},
'cmd_152': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_152.output'),
'state': {'begin': 1666782774.2401013,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.518940925598145,
'end': 1666782785.7590423,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_152.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_152.state')},
'cmd_153': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789875,\n'
' "timestamp_ms" : '
'1666789875781,\n'
' "time" : "Wed Oct 26 15:11:15 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1068184064,\n'
' "io_kbytes" : 1043148,\n'
' "bw_bytes" : 152575926,\n'
' "bw" : 148999,\n'
' "iops" : 297999.714327,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2086296,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1723,\n'
' "max" : 23454,\n'
' "mean" : 1919.903935,\n'
' "stddev" : '
'128.714954,\n'
' "N" : 2086296\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 250,\n'
' "max" : 98895,\n'
' "mean" : 4553.953938,\n'
' "stddev" : '
'1784.346449,\n'
' "N" : 2086297,\n'
' "percentile" : {\n'
' "1.000000" : 3856,\n'
' "5.000000" : 4128,\n'
' "10.000000" : 4192,\n'
' "20.000000" : 4192,\n'
' "30.000000" : 4256,\n'
' "40.000000" : 4256,\n'
' "50.000000" : 4320,\n'
' "60.000000" : 4320,\n'
' "70.000000" : 4384,\n'
' "80.000000" : 4384,\n'
' "90.000000" : 4448,\n'
' "95.000000" : 4640,\n'
' "99.000000" : '
'12608,\n'
' "99.500000" : '
'17536,\n'
' "99.900000" : '
'21888,\n'
' "99.950000" : '
'22144,\n'
' "99.990000" : 62720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5169,\n'
' "max" : 100759,\n'
' "mean" : 6473.857861,\n'
' "stddev" : '
'1787.577180,\n'
' "N" : 2086297\n'
' },\n'
' "bw_min" : 148578,\n'
' "bw_max" : 149622,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'149027.000000,\n'
' "bw_dev" : 264.555069,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 297156,\n'
' "iops_max" : 299244,\n'
' "iops_mean" : '
'298054.000000,\n'
' "iops_stddev" : '
'529.110138,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 48.900000,\n'
' "sys_cpu" : 51.085714,\n'
' "ctx" : 15,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 3.277243,\n'
' "10" : 94.723520,\n'
' "20" : 1.731921,\n'
' "50" : 0.245459,\n'
' "100" : 0.021665,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=7539: Wed '
'Oct 26 15:11:15 2022\n'
' read: IOPS=298k, BW=146MiB/s '
'(153MB/s)(1019MiB/7001msec)\n'
' slat (nsec): min=1723, '
'max=23454, avg=1919.90, '
'stdev=128.71\n'
' clat (nsec): min=250, '
'max=98895, avg=4553.95, '
'stdev=1784.35\n'
' lat (usec): min=5, max=100, '
'avg= 6.47, stdev= 1.79\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3856], '
'5.00th=[ 4128], 10.00th=[ 4192], '
'20.00th=[ 4192],\n'
' | 30.00th=[ 4256], '
'40.00th=[ 4256], 50.00th=[ '
'4320], 60.00th=[ 4320],\n'
' | 70.00th=[ 4384], '
'80.00th=[ 4384], 90.00th=[ '
'4448], 95.00th=[ 4640],\n'
' | 99.00th=[12608], '
'99.50th=[17536], '
'99.90th=[21888], '
'99.95th=[22144],\n'
' | 99.99th=[62720]\n'
' bw ( KiB/s): min=148578, '
'max=149622, per=100.00%, '
'avg=149027.00, stdev=264.56, '
'samples=14\n'
' iops : min=297156, '
'max=299244, avg=298054.00, '
'stdev=529.11, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=3.28%, '
'10=94.72%, 20=1.73%, 50=0.25%, '
'100=0.02%\n'
' cpu : usr=48.90%, '
'sys=51.09%, ctx=15, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2086296,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=146MiB/s (153MB/s), '
'146MiB/s-146MiB/s '
'(153MB/s-153MB/s), io=1019MiB '
'(1068MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_153.output'),
'state': {'begin': 1666782785.7616816,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8182222843170166,
'end': 1666782786.5799038,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_153.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_153.state')},
'cmd_154': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_154.output'),
'state': {'begin': 1666782787.6278014,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8002474308013916,
'end': 1666782788.4280488,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_154.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_154.state')},
'cmd_155': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_155.output'),
'state': {'begin': 1666782788.430456,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.187855958938599,
'end': 1666782799.618312,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_155.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_155.state')},
'cmd_156': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789889,\n'
' "timestamp_ms" : '
'1666789889671,\n'
' "time" : "Wed Oct 26 15:11:29 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 755842560,\n'
' "io_kbytes" : 738127,\n'
' "bw_bytes" : 107962085,\n'
' "bw" : 105431,\n'
' "iops" : 210863.305242,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1476254,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1373,\n'
' "max" : 14708,\n'
' "mean" : 1626.049088,\n'
' "stddev" : '
'183.510630,\n'
' "N" : 1476254\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 180,\n'
' "max" : 118923,\n'
' "mean" : 7596.831394,\n'
' "stddev" : '
'2223.151074,\n'
' "N" : 1476255,\n'
' "percentile" : {\n'
' "1.000000" : 5152,\n'
' "5.000000" : 5280,\n'
' "10.000000" : 5344,\n'
' "20.000000" : 5472,\n'
' "30.000000" : 6752,\n'
' "40.000000" : 7264,\n'
' "50.000000" : 7456,\n'
' "60.000000" : 7712,\n'
' "70.000000" : 8032,\n'
' "80.000000" : 9280,\n'
' "90.000000" : 9408,\n'
' "95.000000" : 9536,\n'
' "99.000000" : '
'16768,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'23936,\n'
' "99.950000" : '
'24448,\n'
' "99.990000" : 61184\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5811,\n'
' "max" : 120425,\n'
' "mean" : 9222.880527,\n'
' "stddev" : '
'2251.499860,\n'
' "N" : 1476255\n'
' },\n'
' "bw_min" : 105209,\n'
' "bw_max" : 105680,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'105450.357143,\n'
' "bw_dev" : 127.924020,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 210418,\n'
' "iops_max" : 211362,\n'
' "iops_mean" : '
'210900.785714,\n'
' "iops_stddev" : '
'255.828722,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.514286,\n'
' "sys_cpu" : 51.542857,\n'
' "ctx" : 741997,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.311100,\n'
' "20" : 2.439621,\n'
' "50" : 0.215004,\n'
' "100" : 0.024522,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=7587: Wed '
'Oct 26 15:11:29 2022\n'
' read: IOPS=211k, BW=103MiB/s '
'(108MB/s)(721MiB/7001msec)\n'
' slat (nsec): min=1373, '
'max=14708, avg=1626.05, '
'stdev=183.51\n'
' clat (nsec): min=180, '
'max=118923, avg=7596.83, '
'stdev=2223.15\n'
' lat (usec): min=5, max=120, '
'avg= 9.22, stdev= 2.25\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5152], '
'5.00th=[ 5280], 10.00th=[ 5344], '
'20.00th=[ 5472],\n'
' | 30.00th=[ 6752], '
'40.00th=[ 7264], 50.00th=[ '
'7456], 60.00th=[ 7712],\n'
' | 70.00th=[ 8032], '
'80.00th=[ 9280], 90.00th=[ '
'9408], 95.00th=[ 9536],\n'
' | 99.00th=[16768], '
'99.50th=[18560], '
'99.90th=[23936], '
'99.95th=[24448],\n'
' | 99.99th=[61184]\n'
' bw ( KiB/s): min=105209, '
'max=105680, per=100.00%, '
'avg=105450.36, stdev=127.92, '
'samples=14\n'
' iops : min=210418, '
'max=211362, avg=210900.79, '
'stdev=255.83, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=97.31%, 20=2.44%, 50=0.22%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=14.51%, '
'sys=51.54%, ctx=741997, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1476254,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=103MiB/s (108MB/s), '
'103MiB/s-103MiB/s '
'(108MB/s-108MB/s), io=721MiB '
'(756MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_156.output'),
'state': {'begin': 1666782799.6249485,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.805732011795044,
'end': 1666782800.4306805,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_156.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_156.state')},
'cmd_157': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_157.output'),
'state': {'begin': 1666782801.4764647,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7939236164093018,
'end': 1666782802.2703884,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_157.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_157.state')},
'cmd_158': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_158.output'),
'state': {'begin': 1666782802.2725344,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.128751754760742,
'end': 1666782813.4012861,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_158.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_158.state')},
'cmd_159': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789903,\n'
' "timestamp_ms" : '
'1666789903447,\n'
' "time" : "Wed Oct 26 15:11:43 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 751161856,\n'
' "io_kbytes" : 733556,\n'
' "bw_bytes" : 107293508,\n'
' "bw" : 104778,\n'
' "iops" : 209557.491787,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1467112,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1402,\n'
' "max" : 30337,\n'
' "mean" : 1675.573352,\n'
' "stddev" : '
'230.937066,\n'
' "N" : 1467112\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2745,\n'
' "max" : 119153,\n'
' "mean" : 7591.607664,\n'
' "stddev" : '
'2285.968158,\n'
' "N" : 1467113,\n'
' "percentile" : {\n'
' "1.000000" : 5216,\n'
' "5.000000" : 5280,\n'
' "10.000000" : 5344,\n'
' "20.000000" : 5472,\n'
' "30.000000" : 5984,\n'
' "40.000000" : 7136,\n'
' "50.000000" : 7456,\n'
' "60.000000" : 7712,\n'
' "70.000000" : 8160,\n'
' "80.000000" : 9408,\n'
' "90.000000" : 9664,\n'
' "95.000000" : 9792,\n'
' "99.000000" : '
'16768,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'24192,\n'
' "99.950000" : '
'24448,\n'
' "99.990000" : 64256\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5801,\n'
' "max" : 120656,\n'
' "mean" : 9267.180865,\n'
' "stddev" : '
'2311.110610,\n'
' "N" : 1467113\n'
' },\n'
' "bw_min" : 104352,\n'
' "bw_max" : 105176,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'104797.642857,\n'
' "bw_dev" : 211.042689,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 208705,\n'
' "iops_max" : 210352,\n'
' "iops_mean" : '
'209595.357143,\n'
' "iops_stddev" : '
'421.922998,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.942857,\n'
' "sys_cpu" : 51.414286,\n'
' "ctx" : 737441,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010156,\n'
' "10" : 97.018701,\n'
' "20" : 2.717039,\n'
' "50" : 0.231816,\n'
' "100" : 0.022016,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=7635: Wed '
'Oct 26 15:11:43 2022\n'
' read: IOPS=210k, BW=102MiB/s '
'(107MB/s)(716MiB/7001msec)\n'
' slat (nsec): min=1402, '
'max=30337, avg=1675.57, '
'stdev=230.94\n'
' clat (usec): min=2, max=119, '
'avg= 7.59, stdev= 2.29\n'
' lat (usec): min=5, max=120, '
'avg= 9.27, stdev= 2.31\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5216], '
'5.00th=[ 5280], 10.00th=[ 5344], '
'20.00th=[ 5472],\n'
' | 30.00th=[ 5984], '
'40.00th=[ 7136], 50.00th=[ '
'7456], 60.00th=[ 7712],\n'
' | 70.00th=[ 8160], '
'80.00th=[ 9408], 90.00th=[ '
'9664], 95.00th=[ 9792],\n'
' | 99.00th=[16768], '
'99.50th=[18560], '
'99.90th=[24192], '
'99.95th=[24448],\n'
' | 99.99th=[64256]\n'
' bw ( KiB/s): min=104352, '
'max=105176, per=100.00%, '
'avg=104797.64, stdev=211.04, '
'samples=14\n'
' iops : min=208705, '
'max=210352, avg=209595.36, '
'stdev=421.92, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=97.02%, 20=2.72%, 50=0.23%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=15.94%, '
'sys=51.41%, ctx=737441, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1467112,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=102MiB/s (107MB/s), '
'102MiB/s-102MiB/s '
'(107MB/s-107MB/s), io=716MiB '
'(751MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_159.output'),
'state': {'begin': 1666782813.4044902,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8058393001556396,
'end': 1666782814.2103295,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_159.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_159.state')},
'cmd_16': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_16.output'),
'state': {'begin': 1666782115.5445716,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8334617614746094,
'end': 1666782116.3780334,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_16.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_16.state')},
'cmd_160': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_io_uring-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_160.output'),
'state': {'begin': 1666782815.1958127,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8335492610931396,
'end': 1666782816.029362,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_160.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_160.state')},
'cmd_161': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_161.output'),
'state': {'begin': 1666782816.03119,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.07723879814148,
'end': 1666782827.1084287,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_161.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_161.state')},
'cmd_162': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789917,\n'
' "timestamp_ms" : '
'1666789917192,\n'
' "time" : "Wed Oct 26 15:11:57 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 754146304,\n'
' "io_kbytes" : 736471,\n'
' "bw_bytes" : 107735186,\n'
' "bw" : 105210,\n'
' "iops" : 210420.142857,\n'
' "runtime" : 7000,\n'
' "total_ios" : 1472941,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1373,\n'
' "max" : 39404,\n'
' "mean" : 1621.156412,\n'
' "stddev" : '
'192.150453,\n'
' "N" : 1472941\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 180,\n'
' "max" : 110587,\n'
' "mean" : 7626.157111,\n'
' "stddev" : '
'2228.400265,\n'
' "N" : 1472942,\n'
' "percentile" : {\n'
' "1.000000" : 5216,\n'
' "5.000000" : 5280,\n'
' "10.000000" : 5344,\n'
' "20.000000" : 5472,\n'
' "30.000000" : 6752,\n'
' "40.000000" : 7264,\n'
' "50.000000" : 7456,\n'
' "60.000000" : 7712,\n'
' "70.000000" : 8096,\n'
' "80.000000" : 9280,\n'
' "90.000000" : 9408,\n'
' "95.000000" : 9664,\n'
' "99.000000" : '
'16768,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'23936,\n'
' "99.950000" : '
'24448,\n'
' "99.990000" : 61696\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5741,\n'
' "max" : 112090,\n'
' "mean" : 9247.313388,\n'
' "stddev" : '
'2260.379617,\n'
' "N" : 1472942\n'
' },\n'
' "bw_min" : 104853,\n'
' "bw_max" : 105510,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'105237.769231,\n'
' "bw_dev" : 207.918716,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 209706,\n'
' "iops_max" : 211020,\n'
' "iops_mean" : '
'210475.769231,\n'
' "iops_stddev" : '
'415.849162,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.157143,\n'
' "sys_cpu" : 51.885714,\n'
' "ctx" : 740379,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.277556,\n'
' "20" : 2.467716,\n'
' "50" : 0.220783,\n'
' "100" : 0.024101,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=7683: Wed '
'Oct 26 15:11:57 2022\n'
' read: IOPS=210k, BW=103MiB/s '
'(108MB/s)(719MiB/7000msec)\n'
' slat (nsec): min=1373, '
'max=39404, avg=1621.16, '
'stdev=192.15\n'
' clat (nsec): min=180, '
'max=110587, avg=7626.16, '
'stdev=2228.40\n'
' lat (usec): min=5, max=112, '
'avg= 9.25, stdev= 2.26\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5216], '
'5.00th=[ 5280], 10.00th=[ 5344], '
'20.00th=[ 5472],\n'
' | 30.00th=[ 6752], '
'40.00th=[ 7264], 50.00th=[ '
'7456], 60.00th=[ 7712],\n'
' | 70.00th=[ 8096], '
'80.00th=[ 9280], 90.00th=[ '
'9408], 95.00th=[ 9664],\n'
' | 99.00th=[16768], '
'99.50th=[18560], '
'99.90th=[23936], '
'99.95th=[24448],\n'
' | 99.99th=[61696]\n'
' bw ( KiB/s): min=104853, '
'max=105510, per=100.00%, '
'avg=105237.77, stdev=207.92, '
'samples=13\n'
' iops : min=209706, '
'max=211020, avg=210475.77, '
'stdev=415.85, samples=13\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=97.28%, 20=2.47%, 50=0.22%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=14.16%, '
'sys=51.89%, ctx=740379, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1472941,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=103MiB/s (108MB/s), '
'103MiB/s-103MiB/s '
'(108MB/s-108MB/s), io=719MiB '
'(754MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_162.output'),
'state': {'begin': 1666782827.111502,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8950073719024658,
'end': 1666782828.0065093,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_162.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_162.state')},
'cmd_163': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_libaio-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_163.output'),
'state': {'begin': 1666782829.0462594,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8603544235229492,
'end': 1666782829.9066138,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_163.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_163.state')},
'cmd_164': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_164.output'),
'state': {'begin': 1666782829.9089046,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.068029165267944,
'end': 1666782840.9769337,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_164.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_164.state')},
'cmd_165': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789931,\n'
' "timestamp_ms" : '
'1666789931060,\n'
' "time" : "Wed Oct 26 15:12:11 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 799335936,\n'
' "io_kbytes" : 780601,\n'
' "bw_bytes" : 114174537,\n'
' "bw" : 111498,\n'
' "iops" : 222997.000429,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1561202,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1442,\n'
' "max" : 21661,\n'
' "mean" : 1649.153308,\n'
' "stddev" : '
'129.768110,\n'
' "N" : 1561202\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2645,\n'
' "max" : 100528,\n'
' "mean" : 7067.635010,\n'
' "stddev" : '
'1738.217242,\n'
' "N" : 1561203,\n'
' "percentile" : {\n'
' "1.000000" : 5856,\n'
' "5.000000" : 5920,\n'
' "10.000000" : 5984,\n'
' "20.000000" : 6112,\n'
' "30.000000" : 6880,\n'
' "40.000000" : 6944,\n'
' "50.000000" : 7008,\n'
' "60.000000" : 7136,\n'
' "70.000000" : 7200,\n'
' "80.000000" : 7264,\n'
' "90.000000" : 7392,\n'
' "95.000000" : 7520,\n'
' "99.000000" : '
'15936,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'25216,\n'
' "99.950000" : '
'25728,\n'
' "99.990000" : 62720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6422,\n'
' "max" : 102221,\n'
' "mean" : 8716.788250,\n'
' "stddev" : '
'1739.886131,\n'
' "N" : 1561203\n'
' },\n'
' "bw_min" : 110982,\n'
' "bw_max" : 112688,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'111528.500000,\n'
' "bw_dev" : 424.069797,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 221965,\n'
' "iops_max" : 225376,\n'
' "iops_mean" : '
'223057.071429,\n'
' "iops_stddev" : '
'848.040498,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 20.328571,\n'
' "sys_cpu" : 47.114286,\n'
' "ctx" : 783978,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.719770,\n'
' "20" : 1.871763,\n'
' "50" : 0.379771,\n'
' "100" : 0.021009,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2224647,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 10878,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 10878,\n'
' "util" : 99.069123\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=7730: Wed '
'Oct 26 15:12:11 2022\n'
' read: IOPS=223k, BW=109MiB/s '
'(114MB/s)(762MiB/7001msec)\n'
' slat (nsec): min=1442, '
'max=21661, avg=1649.15, '
'stdev=129.77\n'
' clat (usec): min=2, max=100, '
'avg= 7.07, stdev= 1.74\n'
' lat (usec): min=6, max=102, '
'avg= 8.72, stdev= 1.74\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5856], '
'5.00th=[ 5920], 10.00th=[ 5984], '
'20.00th=[ 6112],\n'
' | 30.00th=[ 6880], '
'40.00th=[ 6944], 50.00th=[ '
'7008], 60.00th=[ 7136],\n'
' | 70.00th=[ 7200], '
'80.00th=[ 7264], 90.00th=[ '
'7392], 95.00th=[ 7520],\n'
' | 99.00th=[15936], '
'99.50th=[18560], '
'99.90th=[25216], '
'99.95th=[25728],\n'
' | 99.99th=[62720]\n'
' bw ( KiB/s): min=110982, '
'max=112688, per=100.00%, '
'avg=111528.50, stdev=424.07, '
'samples=14\n'
' iops : min=221965, '
'max=225376, avg=223057.07, '
'stdev=848.04, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=97.72%, 20=1.87%, 50=0.38%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=20.33%, '
'sys=47.11%, ctx=783978, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1561202,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=109MiB/s (114MB/s), '
'109MiB/s-109MiB/s '
'(114MB/s-114MB/s), io=762MiB '
'(799MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2224647/0, '
'merge=0/0, ticks=10878/0, '
'in_queue=10878, util=99.07%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_165.output'),
'state': {'begin': 1666782840.9795122,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7958464622497559,
'end': 1666782841.7753587,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_165.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_165.state')},
'cmd_166': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_libaio-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_166.output'),
'state': {'begin': 1666782842.787576,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8798675537109375,
'end': 1666782843.6674435,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_166.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_166.state')},
'cmd_167': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_167.output'),
'state': {'begin': 1666782843.6697602,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.101510524749756,
'end': 1666782854.7712708,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_167.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_167.state')},
'cmd_168': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789944,\n'
' "timestamp_ms" : '
'1666789944855,\n'
' "time" : "Wed Oct 26 15:12:24 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 815441920,\n'
' "io_kbytes" : 796330,\n'
' "bw_bytes" : 116475063,\n'
' "bw" : 113745,\n'
' "iops" : 227490.072847,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1592658,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1472,\n'
' "max" : 20098,\n'
' "mean" : 1679.534212,\n'
' "stddev" : '
'135.170532,\n'
' "N" : 1592659\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 431,\n'
' "max" : 91642,\n'
' "mean" : 6852.600837,\n'
' "stddev" : '
'1799.750306,\n'
' "N" : 1592659,\n'
' "percentile" : {\n'
' "1.000000" : 5856,\n'
' "5.000000" : 5920,\n'
' "10.000000" : 5920,\n'
' "20.000000" : 6048,\n'
' "30.000000" : 6112,\n'
' "40.000000" : 6240,\n'
' "50.000000" : 6368,\n'
' "60.000000" : 7008,\n'
' "70.000000" : 7136,\n'
' "80.000000" : 7328,\n'
' "90.000000" : 7456,\n'
' "95.000000" : 7584,\n'
' "99.000000" : '
'15680,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'25472,\n'
' "99.950000" : '
'25984,\n'
' "99.990000" : 62720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5681,\n'
' "max" : 93386,\n'
' "mean" : 8532.135048,\n'
' "stddev" : '
'1803.334792,\n'
' "N" : 1592659\n'
' },\n'
' "bw_min" : 113374,\n'
' "bw_max" : 114111,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'113776.000000,\n'
' "bw_dev" : 230.273082,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 226748,\n'
' "iops_max" : 228222,\n'
' "iops_mean" : '
'227551.928571,\n'
' "iops_stddev" : '
'460.425633,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 23.000000,\n'
' "sys_cpu" : 47.142857,\n'
' "ctx" : 800025,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.734856,\n'
' "20" : 1.887411,\n'
' "50" : 0.346276,\n'
' "100" : 0.023420,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2265813,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 11215,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 11215,\n'
' "util" : 99.069123\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=7778: Wed '
'Oct 26 15:12:24 2022\n'
' read: IOPS=227k, BW=111MiB/s '
'(116MB/s)(778MiB/7001msec)\n'
' slat (nsec): min=1472, '
'max=20098, avg=1679.53, '
'stdev=135.17\n'
' clat (nsec): min=431, '
'max=91642, avg=6852.60, '
'stdev=1799.75\n'
' lat (nsec): min=5681, '
'max=93386, avg=8532.14, '
'stdev=1803.33\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5856], '
'5.00th=[ 5920], 10.00th=[ 5920], '
'20.00th=[ 6048],\n'
' | 30.00th=[ 6112], '
'40.00th=[ 6240], 50.00th=[ '
'6368], 60.00th=[ 7008],\n'
' | 70.00th=[ 7136], '
'80.00th=[ 7328], 90.00th=[ '
'7456], 95.00th=[ 7584],\n'
' | 99.00th=[15680], '
'99.50th=[18560], '
'99.90th=[25472], '
'99.95th=[25984],\n'
' | 99.99th=[62720]\n'
' bw ( KiB/s): min=113374, '
'max=114111, per=100.00%, '
'avg=113776.00, stdev=230.27, '
'samples=14\n'
' iops : min=226748, '
'max=228222, avg=227551.93, '
'stdev=460.43, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=97.73%, 20=1.89%, 50=0.35%, '
'100=0.02%\n'
' cpu : usr=23.00%, '
'sys=47.14%, ctx=800025, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1592658,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=111MiB/s (116MB/s), '
'111MiB/s-111MiB/s '
'(116MB/s-116MB/s), io=778MiB '
'(815MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2265813/0, '
'merge=0/0, ticks=11215/0, '
'in_queue=11215, util=99.07%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_168.output'),
'state': {'begin': 1666782854.7738547,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8583071231842041,
'end': 1666782855.6321619,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_168.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_168.state')},
'cmd_169': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_libaio-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_169.output'),
'state': {'begin': 1666782856.687171,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.8172504901885986,
'end': 1666782857.5044215,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_169.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_169.state')},
'cmd_17': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_17.output'),
'state': {'begin': 1666782116.380387,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.070308208465576,
'end': 1666782127.4506953,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_17.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_17.state')},
'cmd_170': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_170.output'),
'state': {'begin': 1666782857.506634,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.087865829467773,
'end': 1666782868.5944998,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_170.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_170.state')},
'cmd_171': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789958,\n'
' "timestamp_ms" : '
'1666789958677,\n'
' "time" : "Wed Oct 26 15:12:38 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 812047872,\n'
' "io_kbytes" : 793015,\n'
' "bw_bytes" : 115990268,\n'
' "bw" : 113271,\n'
' "iops" : 226543.208113,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1586029,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1452,\n'
' "max" : 14798,\n'
' "mean" : 1655.304542,\n'
' "stddev" : '
'123.996712,\n'
' "N" : 1586030\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 421,\n'
' "max" : 120085,\n'
' "mean" : 6916.411781,\n'
' "stddev" : '
'1733.488478,\n'
' "N" : 1586030,\n'
' "percentile" : {\n'
' "1.000000" : 5792,\n'
' "5.000000" : 5856,\n'
' "10.000000" : 5920,\n'
' "20.000000" : 6048,\n'
' "30.000000" : 6112,\n'
' "40.000000" : 6816,\n'
' "50.000000" : 6944,\n'
' "60.000000" : 7008,\n'
' "70.000000" : 7136,\n'
' "80.000000" : 7264,\n'
' "90.000000" : 7328,\n'
' "95.000000" : 7520,\n'
' "99.000000" : '
'15680,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'24192,\n'
' "99.950000" : '
'25728,\n'
' "99.990000" : 61184\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5661,\n'
' "max" : 121798,\n'
' "mean" : 8571.716323,\n'
' "stddev" : '
'1737.367517,\n'
' "N" : 1586030\n'
' },\n'
' "bw_min" : 112211,\n'
' "bw_max" : 114174,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'113343.571429,\n'
' "bw_dev" : 546.633292,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 224422,\n'
' "iops_max" : 228348,\n'
' "iops_mean" : '
'226687.285714,\n'
' "iops_stddev" : '
'1093.360904,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 21.314286,\n'
' "sys_cpu" : 47.257143,\n'
' "ctx" : 796376,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.729802,\n'
' "20" : 1.891264,\n'
' "50" : 0.351885,\n'
' "100" : 0.018789,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2250040,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 11049,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 11049,\n'
' "util" : 99.108381\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=7824: Wed '
'Oct 26 15:12:38 2022\n'
' read: IOPS=227k, BW=111MiB/s '
'(116MB/s)(774MiB/7001msec)\n'
' slat (nsec): min=1452, '
'max=14798, avg=1655.30, '
'stdev=124.00\n'
' clat (nsec): min=421, '
'max=120085, avg=6916.41, '
'stdev=1733.49\n'
' lat (usec): min=5, max=121, '
'avg= 8.57, stdev= 1.74\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5792], '
'5.00th=[ 5856], 10.00th=[ 5920], '
'20.00th=[ 6048],\n'
' | 30.00th=[ 6112], '
'40.00th=[ 6816], 50.00th=[ '
'6944], 60.00th=[ 7008],\n'
' | 70.00th=[ 7136], '
'80.00th=[ 7264], 90.00th=[ '
'7328], 95.00th=[ 7520],\n'
' | 99.00th=[15680], '
'99.50th=[18560], '
'99.90th=[24192], '
'99.95th=[25728],\n'
' | 99.99th=[61184]\n'
' bw ( KiB/s): min=112211, '
'max=114174, per=100.00%, '
'avg=113343.57, stdev=546.63, '
'samples=14\n'
' iops : min=224422, '
'max=228348, avg=226687.29, '
'stdev=1093.36, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=97.73%, 20=1.89%, 50=0.35%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=21.31%, '
'sys=47.26%, ctx=796376, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1586029,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=111MiB/s (116MB/s), '
'111MiB/s-111MiB/s '
'(116MB/s-116MB/s), io=774MiB '
'(812MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2250040/0, '
'merge=0/0, ticks=11049/0, '
'in_queue=11049, util=99.11%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_171.output'),
'state': {'begin': 1666782868.597043,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.8535284996032715,
'end': 1666782869.4505715,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_171.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_171.state')},
'cmd_172': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_libaio-bdev_aio_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_172.output'),
'state': {'begin': 1666782871.6327507,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.939302921295166,
'end': 1666782872.5720537,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_172.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_172.state')},
'cmd_173': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_173.output'),
'state': {'begin': 1666782872.5743666,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-bdev_aio_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.872151613235474,
'end': 1666782884.4465182,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_173.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_173.state')},
'cmd_174': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789974,\n'
' "timestamp_ms" : '
'1666789974420,\n'
' "time" : "Wed Oct 26 15:12:54 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1065744896,\n'
' "io_kbytes" : 1040766,\n'
' "bw_bytes" : 152227524,\n'
' "bw" : 148659,\n'
' "iops" : 297319.240109,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2081532,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1663,\n'
' "max" : 22833,\n'
' "mean" : 1889.934944,\n'
' "stddev" : '
'127.676401,\n'
' "N" : 2081532\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 241,\n'
' "max" : 102011,\n'
' "mean" : 4605.353980,\n'
' "stddev" : '
'1741.728440,\n'
' "N" : 2081533,\n'
' "percentile" : {\n'
' "1.000000" : 3952,\n'
' "5.000000" : 4192,\n'
' "10.000000" : 4256,\n'
' "20.000000" : 4256,\n'
' "30.000000" : 4320,\n'
' "40.000000" : 4320,\n'
' "50.000000" : 4384,\n'
' "60.000000" : 4384,\n'
' "70.000000" : 4384,\n'
' "80.000000" : 4448,\n'
' "90.000000" : 4512,\n'
' "95.000000" : 4704,\n'
' "99.000000" : '
'12608,\n'
' "99.500000" : '
'17536,\n'
' "99.900000" : '
'21888,\n'
' "99.950000" : '
'22400,\n'
' "99.990000" : 59648\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5129,\n'
' "max" : 103885,\n'
' "mean" : 6495.288902,\n'
' "stddev" : '
'1745.563229,\n'
' "N" : 2081533\n'
' },\n'
' "bw_min" : 147611,\n'
' "bw_max" : 149598,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'148687.142857,\n'
' "bw_dev" : 617.469882,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 295222,\n'
' "iops_max" : 299196,\n'
' "iops_mean" : '
'297374.357143,\n'
' "iops_stddev" : '
'1234.727102,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 63.985714,\n'
' "sys_cpu" : 35.985714,\n'
' "ctx" : 15,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 99.989959,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 2.077076,\n'
' "10" : 95.930641,\n'
' "20" : 1.729688,\n'
' "50" : 0.242177,\n'
' "100" : 0.017968,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=7881: Wed '
'Oct 26 15:12:54 2022\n'
' read: IOPS=297k, BW=145MiB/s '
'(152MB/s)(1016MiB/7001msec)\n'
' slat (nsec): min=1663, '
'max=22833, avg=1889.93, '
'stdev=127.68\n'
' clat (nsec): min=241, '
'max=102011, avg=4605.35, '
'stdev=1741.73\n'
' lat (usec): min=5, max=103, '
'avg= 6.50, stdev= 1.75\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3952], '
'5.00th=[ 4192], 10.00th=[ 4256], '
'20.00th=[ 4256],\n'
' | 30.00th=[ 4320], '
'40.00th=[ 4320], 50.00th=[ '
'4384], 60.00th=[ 4384],\n'
' | 70.00th=[ 4384], '
'80.00th=[ 4448], 90.00th=[ '
'4512], 95.00th=[ 4704],\n'
' | 99.00th=[12608], '
'99.50th=[17536], '
'99.90th=[21888], '
'99.95th=[22400],\n'
' | 99.99th=[59648]\n'
' bw ( KiB/s): min=147611, '
'max=149598, per=100.00%, '
'avg=148687.14, stdev=617.47, '
'samples=14\n'
' iops : min=295222, '
'max=299196, avg=297374.36, '
'stdev=1234.73, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=2.08%, '
'10=95.93%, 20=1.73%, 50=0.24%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=63.99%, '
'sys=35.99%, ctx=15, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2081532,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=145MiB/s (152MB/s), '
'145MiB/s-145MiB/s '
'(152MB/s-152MB/s), io=1016MiB '
'(1066MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_174.output'),
'state': {'begin': 1666782884.4493692,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.8834080696105957,
'end': 1666782885.3327773,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_174.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_174.state')},
'cmd_175': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_libaio-bdev_aio_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_175.output'),
'state': {'begin': 1666782887.4688983,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.8365371227264404,
'end': 1666782888.3054354,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_175.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_175.state')},
'cmd_176': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_176.output'),
'state': {'begin': 1666782888.307591,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-bdev_aio_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.422178983688354,
'end': 1666782899.72977,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_176.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_176.state')},
'cmd_177': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789989,\n'
' "timestamp_ms" : '
'1666789989718,\n'
' "time" : "Wed Oct 26 15:13:09 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1061161472,\n'
' "io_kbytes" : 1036290,\n'
' "bw_bytes" : 151572842,\n'
' "bw" : 148020,\n'
' "iops" : 296040.422797,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2072579,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1683,\n'
' "max" : 35486,\n'
' "mean" : 1927.547332,\n'
' "stddev" : '
'125.332926,\n'
' "N" : 2072580\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 240,\n'
' "max" : 98986,\n'
' "mean" : 4604.571840,\n'
' "stddev" : '
'1756.957642,\n'
' "N" : 2072579,\n'
' "percentile" : {\n'
' "1.000000" : 3952,\n'
' "5.000000" : 4192,\n'
' "10.000000" : 4256,\n'
' "20.000000" : 4256,\n'
' "30.000000" : 4320,\n'
' "40.000000" : 4320,\n'
' "50.000000" : 4384,\n'
' "60.000000" : 4384,\n'
' "70.000000" : 4384,\n'
' "80.000000" : 4448,\n'
' "90.000000" : 4512,\n'
' "95.000000" : 4704,\n'
' "99.000000" : '
'12608,\n'
' "99.500000" : '
'17536,\n'
' "99.900000" : '
'21888,\n'
' "99.950000" : '
'22400,\n'
' "99.990000" : 61184\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5180,\n'
' "max" : 100919,\n'
' "mean" : 6532.119198,\n'
' "stddev" : '
'1759.977965,\n'
' "N" : 2072579\n'
' },\n'
' "bw_min" : 147349,\n'
' "bw_max" : 148689,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'148046.357143,\n'
' "bw_dev" : 438.091070,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 294698,\n'
' "iops_max" : 297378,\n'
' "iops_mean" : '
'296092.714286,\n'
' "iops_stddev" : '
'876.182140,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 63.528571,\n'
' "sys_cpu" : 36.442857,\n'
' "ctx" : 11,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 99.989240,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 2.120884,\n'
' "10" : 95.873450,\n'
' "20" : 1.738317,\n'
' "50" : 0.244237,\n'
' "100" : 0.020506,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=7940: Wed '
'Oct 26 15:13:09 2022\n'
' read: IOPS=296k, BW=145MiB/s '
'(152MB/s)(1012MiB/7001msec)\n'
' slat (nsec): min=1683, '
'max=35486, avg=1927.55, '
'stdev=125.33\n'
' clat (nsec): min=240, '
'max=98986, avg=4604.57, '
'stdev=1756.96\n'
' lat (usec): min=5, max=100, '
'avg= 6.53, stdev= 1.76\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3952], '
'5.00th=[ 4192], 10.00th=[ 4256], '
'20.00th=[ 4256],\n'
' | 30.00th=[ 4320], '
'40.00th=[ 4320], 50.00th=[ '
'4384], 60.00th=[ 4384],\n'
' | 70.00th=[ 4384], '
'80.00th=[ 4448], 90.00th=[ '
'4512], 95.00th=[ 4704],\n'
' | 99.00th=[12608], '
'99.50th=[17536], '
'99.90th=[21888], '
'99.95th=[22400],\n'
' | 99.99th=[61184]\n'
' bw ( KiB/s): min=147349, '
'max=148689, per=100.00%, '
'avg=148046.36, stdev=438.09, '
'samples=14\n'
' iops : min=294698, '
'max=297378, avg=296092.71, '
'stdev=876.18, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=2.12%, '
'10=95.87%, 20=1.74%, 50=0.24%, '
'100=0.02%\n'
' cpu : usr=63.53%, '
'sys=36.44%, ctx=11, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2072579,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=145MiB/s (152MB/s), '
'145MiB/s-145MiB/s '
'(152MB/s-152MB/s), io=1012MiB '
'(1061MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_177.output'),
'state': {'begin': 1666782899.732175,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.8234589099884033,
'end': 1666782900.555634,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_177.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_177.state')},
'cmd_178': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_libaio-bdev_aio_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_178.output'),
'state': {'begin': 1666782902.744454,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.8046927452087402,
'end': 1666782903.5491467,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_178.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_178.state')},
'cmd_179': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_179.output'),
'state': {'begin': 1666782903.551061,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-bdev_aio_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.563740968704224,
'end': 1666782915.114802,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_179.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_179.state')},
'cmd_18': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789217,\n'
' "timestamp_ms" : '
'1666789217545,\n'
' "time" : "Wed Oct 26 15:00:17 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 392771072,\n'
' "io_kbytes" : 383565,\n'
' "bw_bytes" : 56102138,\n'
' "bw" : 54787,\n'
' "iops" : 109574.489359,\n'
' "runtime" : 7001,\n'
' "total_ios" : 767131,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1423,\n'
' "max" : 5861,\n'
' "mean" : 1520.401970,\n'
' "stddev" : 57.282899,\n'
' "N" : 767131\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1362,\n'
' "max" : 75181,\n'
' "mean" : 7308.434213,\n'
' "stddev" : 994.829189,\n'
' "N" : 767131,\n'
' "percentile" : {\n'
' "1.000000" : 6816,\n'
' "5.000000" : 7008,\n'
' "10.000000" : 7072,\n'
' "20.000000" : 7072,\n'
' "30.000000" : 7136,\n'
' "40.000000" : 7200,\n'
' "50.000000" : 7200,\n'
' "60.000000" : 7264,\n'
' "70.000000" : 7264,\n'
' "80.000000" : 7328,\n'
' "90.000000" : 7456,\n'
' "95.000000" : 7520,\n'
' "99.000000" : 10432,\n'
' "99.500000" : 13632,\n'
' "99.900000" : 15680,\n'
' "99.950000" : 15808,\n'
' "99.990000" : 59136\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6832,\n'
' "max" : 76674,\n'
' "mean" : 8828.836183,\n'
' "stddev" : 997.829487,\n'
' "N" : 767131\n'
' },\n'
' "bw_min" : 54620,\n'
' "bw_max" : 55051,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'54797.071429,\n'
' "bw_dev" : 124.224031,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 109240,\n'
' "iops_max" : 110102,\n'
' "iops_mean" : '
'109594.142857,\n'
' "iops_stddev" : '
'248.448062,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 7.342857,\n'
' "sys_cpu" : 37.014286,\n'
' "ctx" : 767130,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 98.706740,\n'
' "20" : 1.275532,\n'
' "50" : 0.010000,\n'
' "100" : 0.014861,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=5169: Wed '
'Oct 26 15:00:17 2022\n'
' read: IOPS=110k, BW=53.5MiB/s '
'(56.1MB/s)(375MiB/7001msec)\n'
' slat (nsec): min=1423, '
'max=5861, avg=1520.40, '
'stdev=57.28\n'
' clat (nsec): min=1362, '
'max=75181, avg=7308.43, '
'stdev=994.83\n'
' lat (nsec): min=6832, '
'max=76674, avg=8828.84, '
'stdev=997.83\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6816], 5.00th=[ '
'7008], 10.00th=[ 7072], 20.00th=[ '
'7072],\n'
' | 30.00th=[ 7136], 40.00th=[ '
'7200], 50.00th=[ 7200], 60.00th=[ '
'7264],\n'
' | 70.00th=[ 7264], 80.00th=[ '
'7328], 90.00th=[ 7456], 95.00th=[ '
'7520],\n'
' | 99.00th=[10432], '
'99.50th=[13632], 99.90th=[15680], '
'99.95th=[15808],\n'
' | 99.99th=[59136]\n'
' bw ( KiB/s): min=54620, '
'max=55051, per=100.00%, '
'avg=54797.07, stdev=124.22, '
'samples=14\n'
' iops : min=109240, '
'max=110102, avg=109594.14, '
'stdev=248.45, samples=14\n'
' lat (usec) : 2=0.01%, '
'10=98.71%, 20=1.28%, 50=0.01%, '
'100=0.01%\n'
' cpu : usr=7.34%, '
'sys=37.01%, ctx=767130, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=767131,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=53.5MiB/s (56.1MB/s), '
'53.5MiB/s-53.5MiB/s '
'(56.1MB/s-56.1MB/s), io=375MiB '
'(393MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_18.output'),
'state': {'begin': 1666782127.4535794,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8653717041015625,
'end': 1666782128.3189511,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_18.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_18.state')},
'cmd_180': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790005,\n'
' "timestamp_ms" : '
'1666790005075,\n'
' "time" : "Wed Oct 26 15:13:25 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1064973824,\n'
' "io_kbytes" : 1040013,\n'
' "bw_bytes" : 152117386,\n'
' "bw" : 148552,\n'
' "iops" : 297103.985145,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2080025,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1663,\n'
' "max" : 18003,\n'
' "mean" : 1897.384686,\n'
' "stddev" : '
'130.276176,\n'
' "N" : 2080026\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 250,\n'
' "max" : 117160,\n'
' "mean" : 4608.007251,\n'
' "stddev" : '
'1767.883717,\n'
' "N" : 2080025,\n'
' "percentile" : {\n'
' "1.000000" : 3952,\n'
' "5.000000" : 4192,\n'
' "10.000000" : 4256,\n'
' "20.000000" : 4256,\n'
' "30.000000" : 4320,\n'
' "40.000000" : 4320,\n'
' "50.000000" : 4384,\n'
' "60.000000" : 4384,\n'
' "70.000000" : 4448,\n'
' "80.000000" : 4448,\n'
' "90.000000" : 4512,\n'
' "95.000000" : 4704,\n'
' "99.000000" : '
'12608,\n'
' "99.500000" : '
'17536,\n'
' "99.900000" : '
'21888,\n'
' "99.950000" : '
'22400,\n'
' "99.990000" : 61184\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5160,\n'
' "max" : 119033,\n'
' "mean" : 6505.391929,\n'
' "stddev" : '
'1771.826567,\n'
' "N" : 2080025\n'
' },\n'
' "bw_min" : 147892,\n'
' "bw_max" : 149767,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'148627.928571,\n'
' "bw_dev" : 466.567740,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 295785,\n'
' "iops_max" : 299534,\n'
' "iops_mean" : '
'297256.000000,\n'
' "iops_stddev" : '
'932.970030,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 59.700000,\n'
' "sys_cpu" : 40.271429,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 99.989808,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 2.073292,\n'
' "10" : 95.884665,\n'
' "20" : 1.778440,\n'
' "50" : 0.241055,\n'
' "100" : 0.020096,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=7997: Wed '
'Oct 26 15:13:25 2022\n'
' read: IOPS=297k, BW=145MiB/s '
'(152MB/s)(1016MiB/7001msec)\n'
' slat (nsec): min=1663, '
'max=18003, avg=1897.38, '
'stdev=130.28\n'
' clat (nsec): min=250, '
'max=117160, avg=4608.01, '
'stdev=1767.88\n'
' lat (usec): min=5, max=119, '
'avg= 6.51, stdev= 1.77\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3952], '
'5.00th=[ 4192], 10.00th=[ 4256], '
'20.00th=[ 4256],\n'
' | 30.00th=[ 4320], '
'40.00th=[ 4320], 50.00th=[ '
'4384], 60.00th=[ 4384],\n'
' | 70.00th=[ 4448], '
'80.00th=[ 4448], 90.00th=[ '
'4512], 95.00th=[ 4704],\n'
' | 99.00th=[12608], '
'99.50th=[17536], '
'99.90th=[21888], '
'99.95th=[22400],\n'
' | 99.99th=[61184]\n'
' bw ( KiB/s): min=147892, '
'max=149767, per=100.00%, '
'avg=148627.93, stdev=466.57, '
'samples=14\n'
' iops : min=295785, '
'max=299534, avg=297256.00, '
'stdev=932.97, samples=14\n'
' lat (nsec) : 500=0.01%, '
'1000=0.01%\n'
' lat (usec) : 4=2.07%, '
'10=95.88%, 20=1.78%, 50=0.24%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=59.70%, '
'sys=40.27%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2080025,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=145MiB/s (152MB/s), '
'145MiB/s-145MiB/s '
'(152MB/s-152MB/s), io=1016MiB '
'(1065MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_180.output'),
'state': {'begin': 1666782915.1175997,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.8616042137145996,
'end': 1666782915.979204,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_180.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_180.state')},
'cmd_181': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_181.output'),
'state': {'begin': 1666782918.1243794,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8209547996520996,
'end': 1666782918.9453342,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_181.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_181.state')},
'cmd_182': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_182.output'),
'state': {'begin': 1666782918.947441,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.447400093078613,
'end': 1666782930.3948412,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_182.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_182.state')},
'cmd_183': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790020,\n'
' "timestamp_ms" : '
'1666790020382,\n'
' "time" : "Wed Oct 26 15:13:40 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1019141632,\n'
' "io_kbytes" : 995255,\n'
' "bw_bytes" : 145570865,\n'
' "bw" : 142159,\n'
' "iops" : 284317.954578,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1990510,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1693,\n'
' "max" : 23505,\n'
' "mean" : 1928.243363,\n'
' "stddev" : '
'145.937912,\n'
' "N" : 1990510\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 611,\n'
' "max" : 114164,\n'
' "mean" : 4850.434752,\n'
' "stddev" : '
'1749.968240,\n'
' "N" : 1990511,\n'
' "percentile" : {\n'
' "1.000000" : 4256,\n'
' "5.000000" : 4384,\n'
' "10.000000" : 4384,\n'
' "20.000000" : 4512,\n'
' "30.000000" : 4576,\n'
' "40.000000" : 4576,\n'
' "50.000000" : 4640,\n'
' "60.000000" : 4640,\n'
' "70.000000" : 4640,\n'
' "80.000000" : 4704,\n'
' "90.000000" : 4768,\n'
' "95.000000" : 4896,\n'
' "99.000000" : '
'12736,\n'
' "99.500000" : '
'18048,\n'
' "99.900000" : '
'21888,\n'
' "99.950000" : '
'22144,\n'
' "99.990000" : 60672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5460,\n'
' "max" : 116057,\n'
' "mean" : 6778.678102,\n'
' "stddev" : '
'1753.594753,\n'
' "N" : 1990511\n'
' },\n'
' "bw_min" : 141762,\n'
' "bw_max" : 142974,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'142184.785714,\n'
' "bw_dev" : 341.212809,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 283524,\n'
' "iops_max" : 285949,\n'
' "iops_mean" : '
'284369.642857,\n'
' "iops_stddev" : '
'682.603569,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 56.500000,\n'
' "sys_cpu" : 43.471429,\n'
' "ctx" : 7,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 99.977795,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.017684,\n'
' "10" : 97.705714,\n'
' "20" : 1.999236,\n'
' "50" : 0.251091,\n'
' "100" : 0.019744,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=8054: Wed '
'Oct 26 15:13:40 2022\n'
' read: IOPS=284k, BW=139MiB/s '
'(146MB/s)(972MiB/7001msec)\n'
' slat (nsec): min=1693, '
'max=23505, avg=1928.24, '
'stdev=145.94\n'
' clat (nsec): min=611, '
'max=114164, avg=4850.43, '
'stdev=1749.97\n'
' lat (usec): min=5, max=116, '
'avg= 6.78, stdev= 1.75\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4256], '
'5.00th=[ 4384], 10.00th=[ 4384], '
'20.00th=[ 4512],\n'
' | 30.00th=[ 4576], '
'40.00th=[ 4576], 50.00th=[ '
'4640], 60.00th=[ 4640],\n'
' | 70.00th=[ 4640], '
'80.00th=[ 4704], 90.00th=[ '
'4768], 95.00th=[ 4896],\n'
' | 99.00th=[12736], '
'99.50th=[18048], '
'99.90th=[21888], '
'99.95th=[22144],\n'
' | 99.99th=[60672]\n'
' bw ( KiB/s): min=141762, '
'max=142974, per=100.00%, '
'avg=142184.79, stdev=341.21, '
'samples=14\n'
' iops : min=283524, '
'max=285949, avg=284369.64, '
'stdev=682.60, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 4=0.02%, '
'10=97.71%, 20=2.00%, 50=0.25%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=56.50%, '
'sys=43.47%, ctx=7, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1990510,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=139MiB/s (146MB/s), '
'139MiB/s-139MiB/s '
'(146MB/s-146MB/s), io=972MiB '
'(1019MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_183.output'),
'state': {'begin': 1666782930.3976998,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8358349800109863,
'end': 1666782931.2335348,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_183.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_183.state')},
'cmd_184': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_184.output'),
'state': {'begin': 1666782933.2798376,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7966079711914062,
'end': 1666782934.0764456,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_184.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_184.state')},
'cmd_185': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_185.output'),
'state': {'begin': 1666782934.0783417,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.518922328948975,
'end': 1666782945.597264,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_185.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_185.state')},
'cmd_186': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790035,\n'
' "timestamp_ms" : '
'1666790035564,\n'
' "time" : "Wed Oct 26 15:13:55 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1025831424,\n'
' "io_kbytes" : 1001788,\n'
' "bw_bytes" : 146526413,\n'
' "bw" : 143092,\n'
' "iops" : 286184.259392,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2003576,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1713,\n'
' "max" : 22923,\n'
' "mean" : 1939.483181,\n'
' "stddev" : '
'128.999116,\n'
' "N" : 2003576\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 652,\n'
' "max" : 114214,\n'
' "mean" : 4820.917562,\n'
' "stddev" : '
'1748.341345,\n'
' "N" : 2003577,\n'
' "percentile" : {\n'
' "1.000000" : 4256,\n'
' "5.000000" : 4384,\n'
' "10.000000" : 4448,\n'
' "20.000000" : 4512,\n'
' "30.000000" : 4512,\n'
' "40.000000" : 4576,\n'
' "50.000000" : 4576,\n'
' "60.000000" : 4576,\n'
' "70.000000" : 4640,\n'
' "80.000000" : 4640,\n'
' "90.000000" : 4768,\n'
' "95.000000" : 4896,\n'
' "99.000000" : '
'12736,\n'
' "99.500000" : '
'18048,\n'
' "99.900000" : '
'21888,\n'
' "99.950000" : '
'22144,\n'
' "99.990000" : 61184\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5680,\n'
' "max" : 116118,\n'
' "mean" : 6760.400710,\n'
' "stddev" : '
'1752.206198,\n'
' "N" : 2003577\n'
' },\n'
' "bw_min" : 142671,\n'
' "bw_max" : 143940,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'143196.692308,\n'
' "bw_dev" : 456.645629,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 285342,\n'
' "iops_max" : 287880,\n'
' "iops_mean" : '
'286393.538462,\n'
' "iops_stddev" : '
'913.368638,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 55.757143,\n'
' "sys_cpu" : 44.228571,\n'
' "ctx" : 9,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 99.987323,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.023558,\n'
' "10" : 97.738543,\n'
' "20" : 1.962641,\n'
' "50" : 0.251900,\n'
' "100" : 0.019815,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=8112: Wed '
'Oct 26 15:13:55 2022\n'
' read: IOPS=286k, BW=140MiB/s '
'(147MB/s)(978MiB/7001msec)\n'
' slat (nsec): min=1713, '
'max=22923, avg=1939.48, '
'stdev=129.00\n'
' clat (nsec): min=652, '
'max=114214, avg=4820.92, '
'stdev=1748.34\n'
' lat (usec): min=5, max=116, '
'avg= 6.76, stdev= 1.75\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4256], '
'5.00th=[ 4384], 10.00th=[ 4448], '
'20.00th=[ 4512],\n'
' | 30.00th=[ 4512], '
'40.00th=[ 4576], 50.00th=[ '
'4576], 60.00th=[ 4576],\n'
' | 70.00th=[ 4640], '
'80.00th=[ 4640], 90.00th=[ '
'4768], 95.00th=[ 4896],\n'
' | 99.00th=[12736], '
'99.50th=[18048], '
'99.90th=[21888], '
'99.95th=[22144],\n'
' | 99.99th=[61184]\n'
' bw ( KiB/s): min=142671, '
'max=143940, per=100.00%, '
'avg=143196.69, stdev=456.65, '
'samples=13\n'
' iops : min=285342, '
'max=287880, avg=286393.54, '
'stdev=913.37, samples=13\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.02%, 10=97.74%, 20=1.96%, '
'50=0.25%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=55.76%, '
'sys=44.23%, ctx=9, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2003576,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=140MiB/s (147MB/s), '
'140MiB/s-140MiB/s '
'(147MB/s-147MB/s), io=978MiB '
'(1026MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_186.output'),
'state': {'begin': 1666782945.599773,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.804248571395874,
'end': 1666782946.4040215,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_186.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_186.state')},
'cmd_187': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_187.output'),
'state': {'begin': 1666782948.5101879,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7871615886688232,
'end': 1666782949.2973495,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_187.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_187.state')},
'cmd_188': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_188.output'),
'state': {'begin': 1666782949.2993407,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.528333902359009,
'end': 1666782960.8276746,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_188.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_188.state')},
'cmd_189': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790050,\n'
' "timestamp_ms" : '
'1666790050783,\n'
' "time" : "Wed Oct 26 15:14:10 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1025968128,\n'
' "io_kbytes" : 1001922,\n'
' "bw_bytes" : 146566875,\n'
' "bw" : 143131,\n'
' "iops" : 286263.142857,\n'
' "runtime" : 7000,\n'
' "total_ios" : 2003842,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1683,\n'
' "max" : 22953,\n'
' "mean" : 1917.783290,\n'
' "stddev" : '
'143.998960,\n'
' "N" : 2003843\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 651,\n'
' "max" : 119825,\n'
' "mean" : 4827.615465,\n'
' "stddev" : '
'1728.532459,\n'
' "N" : 2003842,\n'
' "percentile" : {\n'
' "1.000000" : 4256,\n'
' "5.000000" : 4384,\n'
' "10.000000" : 4384,\n'
' "20.000000" : 4448,\n'
' "30.000000" : 4512,\n'
' "40.000000" : 4576,\n'
' "50.000000" : 4640,\n'
' "60.000000" : 4640,\n'
' "70.000000" : 4640,\n'
' "80.000000" : 4704,\n'
' "90.000000" : 4768,\n'
' "95.000000" : 4960,\n'
' "99.000000" : '
'12736,\n'
' "99.500000" : '
'18048,\n'
' "99.900000" : '
'21888,\n'
' "99.950000" : '
'22144,\n'
' "99.990000" : 59648\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5410,\n'
' "max" : 121838,\n'
' "mean" : 6745.398757,\n'
' "stddev" : '
'1732.651265,\n'
' "N" : 2003842\n'
' },\n'
' "bw_min" : 142876,\n'
' "bw_max" : 143492,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'143137.428571,\n'
' "bw_dev" : 166.771476,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 285752,\n'
' "iops_max" : 286984,\n'
' "iops_mean" : '
'286274.928571,\n'
' "iops_stddev" : '
'333.438142,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 55.185714,\n'
' "sys_cpu" : 44.785714,\n'
' "ctx" : 7,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 99.978691,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.016418,\n'
' "10" : 97.727366,\n'
' "20" : 1.985785,\n'
' "50" : 0.247275,\n'
' "100" : 0.017267,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=8172: Wed '
'Oct 26 15:14:10 2022\n'
' read: IOPS=286k, BW=140MiB/s '
'(147MB/s)(978MiB/7000msec)\n'
' slat (nsec): min=1683, '
'max=22953, avg=1917.78, '
'stdev=144.00\n'
' clat (nsec): min=651, '
'max=119825, avg=4827.62, '
'stdev=1728.53\n'
' lat (usec): min=5, max=121, '
'avg= 6.75, stdev= 1.73\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4256], '
'5.00th=[ 4384], 10.00th=[ 4384], '
'20.00th=[ 4448],\n'
' | 30.00th=[ 4512], '
'40.00th=[ 4576], 50.00th=[ '
'4640], 60.00th=[ 4640],\n'
' | 70.00th=[ 4640], '
'80.00th=[ 4704], 90.00th=[ '
'4768], 95.00th=[ 4960],\n'
' | 99.00th=[12736], '
'99.50th=[18048], '
'99.90th=[21888], '
'99.95th=[22144],\n'
' | 99.99th=[59648]\n'
' bw ( KiB/s): min=142876, '
'max=143492, per=100.00%, '
'avg=143137.43, stdev=166.77, '
'samples=14\n'
' iops : min=285752, '
'max=286984, avg=286274.93, '
'stdev=333.44, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 4=0.02%, '
'10=97.73%, 20=1.99%, 50=0.25%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=55.19%, '
'sys=44.79%, ctx=7, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2003842,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=140MiB/s (147MB/s), '
'140MiB/s-140MiB/s '
'(147MB/s-147MB/s), io=978MiB '
'(1026MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_189.output'),
'state': {'begin': 1666782960.8303506,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8077137470245361,
'end': 1666782961.6380644,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_189.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_189.state')},
'cmd_19': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_19.output'),
'state': {'begin': 1666782130.4222083,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.864431619644165,
'end': 1666782131.28664,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_19.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_19.state')},
'cmd_190': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_libaio-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_190.output'),
'state': {'begin': 1666782962.6391451,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8529889583587646,
'end': 1666782963.492134,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_190.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_190.state')},
'cmd_191': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_191.output'),
'state': {'begin': 1666782963.4948034,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.136729001998901,
'end': 1666782974.6315324,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_191.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_191.state')},
'cmd_192': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790064,\n'
' "timestamp_ms" : '
'1666790064713,\n'
' "time" : "Wed Oct 26 15:14:24 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 840821760,\n'
' "io_kbytes" : 821115,\n'
' "bw_bytes" : 120100237,\n'
' "bw" : 117285,\n'
' "iops" : 234570.489930,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1642228,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1503,\n'
' "max" : 27501,\n'
' "mean" : 1736.627195,\n'
' "stddev" : '
'156.192241,\n'
' "N" : 1642229\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2836,\n'
' "max" : 121638,\n'
' "mean" : 6548.460939,\n'
' "stddev" : '
'1775.773390,\n'
' "N" : 1642229,\n'
' "percentile" : {\n'
' "1.000000" : 5920,\n'
' "5.000000" : 5920,\n'
' "10.000000" : 5984,\n'
' "20.000000" : 6048,\n'
' "30.000000" : 6112,\n'
' "40.000000" : 6176,\n'
' "50.000000" : 6240,\n'
' "60.000000" : 6304,\n'
' "70.000000" : 6368,\n'
' "80.000000" : 6432,\n'
' "90.000000" : 7136,\n'
' "95.000000" : 7584,\n'
' "99.000000" : '
'15040,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'25984,\n'
' "99.950000" : '
'26496,\n'
' "99.990000" : 62208\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6603,\n'
' "max" : 123481,\n'
' "mean" : 8285.088133,\n'
' "stddev" : '
'1774.881999,\n'
' "N" : 1642229\n'
' },\n'
' "bw_min" : 116619,\n'
' "bw_max" : 117728,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'117305.857143,\n'
' "bw_dev" : 299.454853,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 233238,\n'
' "iops_max" : 235456,\n'
' "iops_mean" : '
'234611.928571,\n'
' "iops_stddev" : '
'598.818126,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 19.800000,\n'
' "sys_cpu" : 57.285714,\n'
' "ctx" : 824948,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.830691,\n'
' "20" : 1.792078,\n'
' "50" : 0.349099,\n'
' "100" : 0.020764,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=8219: Wed '
'Oct 26 15:14:24 2022\n'
' read: IOPS=235k, BW=115MiB/s '
'(120MB/s)(802MiB/7001msec)\n'
' slat (nsec): min=1503, '
'max=27501, avg=1736.63, '
'stdev=156.19\n'
' clat (usec): min=2, max=121, '
'avg= 6.55, stdev= 1.78\n'
' lat (usec): min=6, max=123, '
'avg= 8.29, stdev= 1.77\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5920], '
'5.00th=[ 5920], 10.00th=[ 5984], '
'20.00th=[ 6048],\n'
' | 30.00th=[ 6112], '
'40.00th=[ 6176], 50.00th=[ '
'6240], 60.00th=[ 6304],\n'
' | 70.00th=[ 6368], '
'80.00th=[ 6432], 90.00th=[ '
'7136], 95.00th=[ 7584],\n'
' | 99.00th=[15040], '
'99.50th=[18560], '
'99.90th=[25984], '
'99.95th=[26496],\n'
' | 99.99th=[62208]\n'
' bw ( KiB/s): min=116619, '
'max=117728, per=100.00%, '
'avg=117305.86, stdev=299.45, '
'samples=14\n'
' iops : min=233238, '
'max=235456, avg=234611.93, '
'stdev=598.82, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=97.83%, 20=1.79%, 50=0.35%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=19.80%, '
'sys=57.29%, ctx=824948, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1642228,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=115MiB/s (120MB/s), '
'115MiB/s-115MiB/s '
'(120MB/s-120MB/s), io=802MiB '
'(841MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_192.output'),
'state': {'begin': 1666782974.6349154,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8530197143554688,
'end': 1666782975.487935,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_192.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_192.state')},
'cmd_193': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_libaio-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_193.output'),
'state': {'begin': 1666782976.5959628,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8007280826568604,
'end': 1666782977.3966908,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_193.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_193.state')},
'cmd_194': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_194.output'),
'state': {'begin': 1666782977.398984,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.133901357650757,
'end': 1666782988.5328853,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_194.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_194.state')},
'cmd_195': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790078,\n'
' "timestamp_ms" : '
'1666790078614,\n'
' "time" : "Wed Oct 26 15:14:38 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 837950976,\n'
' "io_kbytes" : 818311,\n'
' "bw_bytes" : 119690183,\n'
' "bw" : 116884,\n'
' "iops" : 233769.747179,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1636622,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1513,\n'
' "max" : 34655,\n'
' "mean" : 1732.315724,\n'
' "stddev" : '
'166.086574,\n'
' "N" : 1636622\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2925,\n'
' "max" : 119975,\n'
' "mean" : 6582.195259,\n'
' "stddev" : '
'1784.605938,\n'
' "N" : 1636623,\n'
' "percentile" : {\n'
' "1.000000" : 5920,\n'
' "5.000000" : 5984,\n'
' "10.000000" : 5984,\n'
' "20.000000" : 6048,\n'
' "30.000000" : 6112,\n'
' "40.000000" : 6176,\n'
' "50.000000" : 6304,\n'
' "60.000000" : 6368,\n'
' "70.000000" : 6432,\n'
' "80.000000" : 6496,\n'
' "90.000000" : 7072,\n'
' "95.000000" : 7584,\n'
' "99.000000" : '
'15296,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'25984,\n'
' "99.950000" : '
'26496,\n'
' "99.990000" : 62720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5931,\n'
' "max" : 121779,\n'
' "mean" : 8314.510898,\n'
' "stddev" : '
'1783.242623,\n'
' "N" : 1636623\n'
' },\n'
' "bw_min" : 116388,\n'
' "bw_max" : 117454,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'116959.071429,\n'
' "bw_dev" : 338.761878,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 232776,\n'
' "iops_max" : 234908,\n'
' "iops_mean" : '
'233918.357143,\n'
' "iops_stddev" : '
'677.455262,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 20.128571,\n'
' "sys_cpu" : 56.842857,\n'
' "ctx" : 822028,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.817945,\n'
' "20" : 1.790823,\n'
' "50" : 0.360560,\n'
' "100" : 0.022241,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=8268: Wed '
'Oct 26 15:14:38 2022\n'
' read: IOPS=234k, BW=114MiB/s '
'(120MB/s)(799MiB/7001msec)\n'
' slat (nsec): min=1513, '
'max=34655, avg=1732.32, '
'stdev=166.09\n'
' clat (usec): min=2, max=119, '
'avg= 6.58, stdev= 1.78\n'
' lat (usec): min=5, max=121, '
'avg= 8.31, stdev= 1.78\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5920], '
'5.00th=[ 5984], 10.00th=[ 5984], '
'20.00th=[ 6048],\n'
' | 30.00th=[ 6112], '
'40.00th=[ 6176], 50.00th=[ '
'6304], 60.00th=[ 6368],\n'
' | 70.00th=[ 6432], '
'80.00th=[ 6496], 90.00th=[ '
'7072], 95.00th=[ 7584],\n'
' | 99.00th=[15296], '
'99.50th=[18560], '
'99.90th=[25984], '
'99.95th=[26496],\n'
' | 99.99th=[62720]\n'
' bw ( KiB/s): min=116388, '
'max=117454, per=100.00%, '
'avg=116959.07, stdev=338.76, '
'samples=14\n'
' iops : min=232776, '
'max=234908, avg=233918.36, '
'stdev=677.46, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=97.82%, 20=1.79%, 50=0.36%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=20.13%, '
'sys=56.84%, ctx=822028, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1636622,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=114MiB/s (120MB/s), '
'114MiB/s-114MiB/s '
'(120MB/s-120MB/s), io=799MiB '
'(838MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_195.output'),
'state': {'begin': 1666782988.5356045,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8266096115112305,
'end': 1666782989.362214,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_195.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_195.state')},
'cmd_196': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_2_libaio-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_196.output'),
'state': {'begin': 1666782990.3637896,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8320932388305664,
'end': 1666782991.1958828,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_196.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_196.state')},
'cmd_197': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_197.output'),
'state': {'begin': 1666782991.1985276,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.097421169281006,
'end': 1666783002.2959487,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_197.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_197.state')},
'cmd_198': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790092,\n'
' "timestamp_ms" : '
'1666790092370,\n'
' "time" : "Wed Oct 26 15:14:52 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 836393472,\n'
' "io_kbytes" : 816790,\n'
' "bw_bytes" : 119467714,\n'
' "bw" : 116667,\n'
' "iops" : 233335.237823,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1633580,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1512,\n'
' "max" : 53531,\n'
' "mean" : 1744.740057,\n'
' "stddev" : '
'169.297769,\n'
' "N" : 1633580\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2875,\n'
' "max" : 119985,\n'
' "mean" : 6578.702695,\n'
' "stddev" : '
'1794.962820,\n'
' "N" : 1633581,\n'
' "percentile" : {\n'
' "1.000000" : 5920,\n'
' "5.000000" : 5984,\n'
' "10.000000" : 5984,\n'
' "20.000000" : 6048,\n'
' "30.000000" : 6112,\n'
' "40.000000" : 6176,\n'
' "50.000000" : 6304,\n'
' "60.000000" : 6304,\n'
' "70.000000" : 6368,\n'
' "80.000000" : 6496,\n'
' "90.000000" : 7136,\n'
' "95.000000" : 7584,\n'
' "99.000000" : '
'15296,\n'
' "99.500000" : '
'18560,\n'
' "99.900000" : '
'25984,\n'
' "99.950000" : '
'26496,\n'
' "99.990000" : 62720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6793,\n'
' "max" : 121838,\n'
' "mean" : 8323.442636,\n'
' "stddev" : '
'1795.815444,\n'
' "N" : 1633581\n'
' },\n'
' "bw_min" : 115883,\n'
' "bw_max" : 117515,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'116731.142857,\n'
' "bw_dev" : 532.190951,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 231766,\n'
' "iops_max" : 235030,\n'
' "iops_mean" : '
'233462.428571,\n'
' "iops_stddev" : '
'1064.482230,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 20.028571,\n'
' "sys_cpu" : 57.300000,\n'
' "ctx" : 820575,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.820186,\n'
' "20" : 1.795872,\n'
' "50" : 0.354130,\n'
' "100" : 0.022833,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=8316: Wed '
'Oct 26 15:14:52 2022\n'
' read: IOPS=233k, BW=114MiB/s '
'(119MB/s)(798MiB/7001msec)\n'
' slat (nsec): min=1512, '
'max=53531, avg=1744.74, '
'stdev=169.30\n'
' clat (usec): min=2, max=119, '
'avg= 6.58, stdev= 1.79\n'
' lat (usec): min=6, max=121, '
'avg= 8.32, stdev= 1.80\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5920], '
'5.00th=[ 5984], 10.00th=[ 5984], '
'20.00th=[ 6048],\n'
' | 30.00th=[ 6112], '
'40.00th=[ 6176], 50.00th=[ '
'6304], 60.00th=[ 6304],\n'
' | 70.00th=[ 6368], '
'80.00th=[ 6496], 90.00th=[ '
'7136], 95.00th=[ 7584],\n'
' | 99.00th=[15296], '
'99.50th=[18560], '
'99.90th=[25984], '
'99.95th=[26496],\n'
' | 99.99th=[62720]\n'
' bw ( KiB/s): min=115883, '
'max=117515, per=100.00%, '
'avg=116731.14, stdev=532.19, '
'samples=14\n'
' iops : min=231766, '
'max=235030, avg=233462.43, '
'stdev=1064.48, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=97.82%, 20=1.80%, 50=0.35%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=20.03%, '
'sys=57.30%, ctx=820575, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1633580,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=114MiB/s (119MB/s), '
'114MiB/s-114MiB/s '
'(119MB/s-119MB/s), io=798MiB '
'(836MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_198.output'),
'state': {'begin': 1666783002.302555,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8001463413238525,
'end': 1666783003.1027014,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_198.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_198.state')},
'cmd_199': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_199.output'),
'state': {'begin': 1666783004.1544373,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8408300876617432,
'end': 1666783004.9952674,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_199.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_199.state')},
'cmd_20': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_20.output'),
'state': {'begin': 1666782131.2885804,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.547215700149536,
'end': 1666782142.835796,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_20.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_20.state')},
'cmd_200': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_200.output'),
'state': {'begin': 1666783004.997414,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.062525272369385,
'end': 1666783016.0599394,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_200.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_200.state')},
'cmd_201': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790106,\n'
' "timestamp_ms" : '
'1666790106140,\n'
' "time" : "Wed Oct 26 15:15:06 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1580787200,\n'
' "io_kbytes" : 1543737,\n'
' "bw_bytes" : 225794486,\n'
' "bw" : 220502,\n'
' "iops" : 441004.285102,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3087471,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1172,\n'
' "max" : 41017,\n'
' "mean" : 1789.167935,\n'
' "stddev" : '
'375.554215,\n'
' "N" : 3087472\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 100,\n'
' "max" : 122480,\n'
' "mean" : 7011.723363,\n'
' "stddev" : '
'2863.847161,\n'
' "N" : 3087474,\n'
' "percentile" : {\n'
' "1.000000" : 4128,\n'
' "5.000000" : 4704,\n'
' "10.000000" : 4768,\n'
' "20.000000" : 6112,\n'
' "30.000000" : 6304,\n'
' "40.000000" : 6368,\n'
' "50.000000" : 6432,\n'
' "60.000000" : 6624,\n'
' "70.000000" : 6944,\n'
' "80.000000" : 7584,\n'
' "90.000000" : 8512,\n'
' "95.000000" : 9280,\n'
' "99.000000" : '
'20864,\n'
' "99.500000" : '
'25216,\n'
' "99.900000" : '
'27008,\n'
' "99.950000" : '
'33536,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5751,\n'
' "max" : 123712,\n'
' "mean" : 8800.891246,\n'
' "stddev" : '
'2837.715863,\n'
' "N" : 3087474\n'
' },\n'
' "bw_min" : 219155,\n'
' "bw_max" : 221950,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'220618.285714,\n'
' "bw_dev" : 970.234264,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 438310,\n'
' "iops_max" : 443900,\n'
' "iops_mean" : '
'441236.785714,\n'
' "iops_stddev" : '
'1940.453523,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.271429,\n'
' "sys_cpu" : 80.428571,\n'
' "ctx" : 35740,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.016356,\n'
' "10" : 95.917889,\n'
' "20" : 2.579652,\n'
' "50" : 1.446815,\n'
' "100" : 0.039093,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=8364: Wed Oct 26 15:15:06 '
'2022\n'
' read: IOPS=441k, BW=215MiB/s '
'(226MB/s)(1508MiB/7001msec)\n'
' slat (nsec): min=1172, '
'max=41017, avg=1789.17, '
'stdev=375.55\n'
' clat (nsec): min=100, '
'max=122480, avg=7011.72, '
'stdev=2863.85\n'
' lat (usec): min=5, max=123, '
'avg= 8.80, stdev= 2.84\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4128], '
'5.00th=[ 4704], 10.00th=[ 4768], '
'20.00th=[ 6112],\n'
' | 30.00th=[ 6304], '
'40.00th=[ 6368], 50.00th=[ '
'6432], 60.00th=[ 6624],\n'
' | 70.00th=[ 6944], '
'80.00th=[ 7584], 90.00th=[ '
'8512], 95.00th=[ 9280],\n'
' | 99.00th=[20864], '
'99.50th=[25216], '
'99.90th=[27008], '
'99.95th=[33536],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=219155, '
'max=221950, per=100.00%, '
'avg=220618.29, stdev=970.23, '
'samples=14\n'
' iops : min=438310, '
'max=443900, avg=441236.79, '
'stdev=1940.45, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.02%, 10=95.92%, 20=2.58%, '
'50=1.45%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=15.27%, '
'sys=80.43%, ctx=35740, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3087471,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=215MiB/s (226MB/s), '
'215MiB/s-215MiB/s '
'(226MB/s-226MB/s), io=1508MiB '
'(1581MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_201.output'),
'state': {'begin': 1666783016.0625746,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8646080493927002,
'end': 1666783016.9271827,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_201.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_201.state')},
'cmd_202': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_202.output'),
'state': {'begin': 1666783017.9683626,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8255369663238525,
'end': 1666783018.7938995,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_202.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_202.state')},
'cmd_203': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_203.output'),
'state': {'begin': 1666783018.7958016,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.042908668518066,
'end': 1666783029.8387103,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_203.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_203.state')},
'cmd_204': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790119,\n'
' "timestamp_ms" : '
'1666790119919,\n'
' "time" : "Wed Oct 26 15:15:19 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1603376128,\n'
' "io_kbytes" : 1565797,\n'
' "bw_bytes" : 229021015,\n'
' "bw" : 223653,\n'
' "iops" : 447306.099129,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3131590,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1162,\n'
' "max" : 24146,\n'
' "mean" : 1765.691817,\n'
' "stddev" : '
'356.119959,\n'
' "N" : 3131591\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 90,\n'
' "max" : 125896,\n'
' "mean" : 6918.897432,\n'
' "stddev" : '
'2872.697041,\n'
' "N" : 3131593,\n'
' "percentile" : {\n'
' "1.000000" : 4080,\n'
' "5.000000" : 4640,\n'
' "10.000000" : 4704,\n'
' "20.000000" : 6048,\n'
' "30.000000" : 6176,\n'
' "40.000000" : 6240,\n'
' "50.000000" : 6304,\n'
' "60.000000" : 6432,\n'
' "70.000000" : 6816,\n'
' "80.000000" : 7456,\n'
' "90.000000" : 8512,\n'
' "95.000000" : 9152,\n'
' "99.000000" : '
'20864,\n'
' "99.500000" : '
'25216,\n'
' "99.900000" : '
'27008,\n'
' "99.950000" : '
'33024,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5511,\n'
' "max" : 127119,\n'
' "mean" : 8684.589193,\n'
' "stddev" : '
'2830.926192,\n'
' "N" : 3131593\n'
' },\n'
' "bw_min" : 222821,\n'
' "bw_max" : 224344,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'223701.928571,\n'
' "bw_dev" : 419.962354,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 445642,\n'
' "iops_max" : 448688,\n'
' "iops_mean" : '
'447403.571429,\n'
' "iops_stddev" : '
'839.997868,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.257143,\n'
' "sys_cpu" : 78.285714,\n'
' "ctx" : 39299,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.170808,\n'
' "10" : 95.608844,\n'
' "20" : 2.736278,\n'
' "50" : 1.445496,\n'
' "100" : 0.037393,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=8410: Wed Oct 26 15:15:19 '
'2022\n'
' read: IOPS=447k, BW=218MiB/s '
'(229MB/s)(1529MiB/7001msec)\n'
' slat (nsec): min=1162, '
'max=24146, avg=1765.69, '
'stdev=356.12\n'
' clat (nsec): min=90, '
'max=125896, avg=6918.90, '
'stdev=2872.70\n'
' lat (usec): min=5, max=127, '
'avg= 8.68, stdev= 2.83\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4080], '
'5.00th=[ 4640], 10.00th=[ 4704], '
'20.00th=[ 6048],\n'
' | 30.00th=[ 6176], '
'40.00th=[ 6240], 50.00th=[ '
'6304], 60.00th=[ 6432],\n'
' | 70.00th=[ 6816], '
'80.00th=[ 7456], 90.00th=[ '
'8512], 95.00th=[ 9152],\n'
' | 99.00th=[20864], '
'99.50th=[25216], '
'99.90th=[27008], '
'99.95th=[33024],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=222821, '
'max=224344, per=100.00%, '
'avg=223701.93, stdev=419.96, '
'samples=14\n'
' iops : min=445642, '
'max=448688, avg=447403.57, '
'stdev=840.00, samples=14\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.17%, 10=95.61%, 20=2.74%, '
'50=1.45%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=17.26%, '
'sys=78.29%, ctx=39299, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3131590,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=218MiB/s (229MB/s), '
'218MiB/s-218MiB/s '
'(229MB/s-229MB/s), io=1529MiB '
'(1603MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_204.output'),
'state': {'begin': 1666783029.841205,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8257076740264893,
'end': 1666783030.6669126,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_204.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_204.state')},
'cmd_205': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_205.output'),
'state': {'begin': 1666783031.7331214,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.8146262168884277,
'end': 1666783032.5477476,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_205.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_205.state')},
'cmd_206': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_206.output'),
'state': {'begin': 1666783032.5496573,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.072738885879517,
'end': 1666783043.6223962,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_206.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_206.state')},
'cmd_207': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790133,\n'
' "timestamp_ms" : '
'1666790133702,\n'
' "time" : "Wed Oct 26 15:15:33 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1593121280,\n'
' "io_kbytes" : 1555782,\n'
' "bw_bytes" : 227556246,\n'
' "bw" : 222222,\n'
' "iops" : 444445.222111,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3111561,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1162,\n'
' "max" : 18786,\n'
' "mean" : 1779.421280,\n'
' "stddev" : '
'358.143700,\n'
' "N" : 3111562\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 90,\n'
' "max" : 126347,\n'
' "mean" : 6962.273469,\n'
' "stddev" : '
'2892.549055,\n'
' "N" : 3111564,\n'
' "percentile" : {\n'
' "1.000000" : 4128,\n'
' "5.000000" : 4640,\n'
' "10.000000" : 4768,\n'
' "20.000000" : 6112,\n'
' "30.000000" : 6176,\n'
' "40.000000" : 6304,\n'
' "50.000000" : 6368,\n'
' "60.000000" : 6496,\n'
' "70.000000" : 6880,\n'
' "80.000000" : 7456,\n'
' "90.000000" : 8512,\n'
' "95.000000" : 9152,\n'
' "99.000000" : '
'20864,\n'
' "99.500000" : '
'25472,\n'
' "99.900000" : '
'27264,\n'
' "99.950000" : '
'34048,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5530,\n'
' "max" : 127549,\n'
' "mean" : 8741.694742,\n'
' "stddev" : '
'2854.750029,\n'
' "N" : 3111564\n'
' },\n'
' "bw_min" : 221009,\n'
' "bw_max" : 223129,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'222349.285714,\n'
' "bw_dev" : 743.621536,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 442018,\n'
' "iops_max" : 446258,\n'
' "iops_mean" : '
'444698.428571,\n'
' "iops_stddev" : '
'1487.477820,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 16.157143,\n'
' "sys_cpu" : 79.471429,\n'
' "ctx" : 36179,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.077582,\n'
' "10" : 95.868408,\n'
' "20" : 2.476474,\n'
' "50" : 1.537267,\n'
' "100" : 0.039498,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=8456: Wed Oct 26 15:15:33 '
'2022\n'
' read: IOPS=444k, BW=217MiB/s '
'(228MB/s)(1519MiB/7001msec)\n'
' slat (nsec): min=1162, '
'max=18786, avg=1779.42, '
'stdev=358.14\n'
' clat (nsec): min=90, '
'max=126347, avg=6962.27, '
'stdev=2892.55\n'
' lat (usec): min=5, max=127, '
'avg= 8.74, stdev= 2.85\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4128], '
'5.00th=[ 4640], 10.00th=[ 4768], '
'20.00th=[ 6112],\n'
' | 30.00th=[ 6176], '
'40.00th=[ 6304], 50.00th=[ '
'6368], 60.00th=[ 6496],\n'
' | 70.00th=[ 6880], '
'80.00th=[ 7456], 90.00th=[ '
'8512], 95.00th=[ 9152],\n'
' | 99.00th=[20864], '
'99.50th=[25472], '
'99.90th=[27264], '
'99.95th=[34048],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=221009, '
'max=223129, per=100.00%, '
'avg=222349.29, stdev=743.62, '
'samples=14\n'
' iops : min=442018, '
'max=446258, avg=444698.43, '
'stdev=1487.48, samples=14\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.08%, 10=95.87%, 20=2.48%, '
'50=1.54%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=16.16%, '
'sys=79.47%, ctx=36179, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3111561,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=217MiB/s (228MB/s), '
'217MiB/s-217MiB/s '
'(228MB/s-228MB/s), io=1519MiB '
'(1593MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_207.output'),
'state': {'begin': 1666783043.6253064,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7937602996826172,
'end': 1666783044.4190667,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_207.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_207.state')},
'cmd_208': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_208.output'),
'state': {'begin': 1666783045.4686837,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7892277240753174,
'end': 1666783046.2579114,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_208.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_208.state')},
'cmd_209': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_209.output'),
'state': {'begin': 1666783046.2600665,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.105571985244751,
'end': 1666783057.3656385,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_209.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_209.state')},
'cmd_21': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789232,\n'
' "timestamp_ms" : '
'1666789232866,\n'
' "time" : "Wed Oct 26 15:00:32 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 584781824,\n'
' "io_kbytes" : 571076,\n'
' "bw_bytes" : 83528327,\n'
' "bw" : 81570,\n'
' "iops" : 163141.265533,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1142152,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 230,\n'
' "max" : 3617,\n'
' "mean" : 269.127672,\n'
' "stddev" : 21.625019,\n'
' "N" : 1142152\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2775,\n'
' "max" : 111990,\n'
' "mean" : 5644.366970,\n'
' "stddev" : '
'1128.740165,\n'
' "N" : 1142152,\n'
' "percentile" : {\n'
' "1.000000" : 5344,\n'
' "5.000000" : 5344,\n'
' "10.000000" : 5408,\n'
' "20.000000" : 5408,\n'
' "30.000000" : 5472,\n'
' "40.000000" : 5472,\n'
' "50.000000" : 5472,\n'
' "60.000000" : 5536,\n'
' "70.000000" : 5536,\n'
' "80.000000" : 5600,\n'
' "90.000000" : 5664,\n'
' "95.000000" : 5728,\n'
' "99.000000" : 11456,\n'
' "99.500000" : 12480,\n'
' "99.900000" : 16768,\n'
' "99.950000" : 17024,\n'
' "99.990000" : 58624\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5560,\n'
' "max" : 112270,\n'
' "mean" : 5913.494643,\n'
' "stddev" : '
'1129.505716,\n'
' "N" : 1142152\n'
' },\n'
' "bw_min" : 81452,\n'
' "bw_max" : 81832,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'81605.142857,\n'
' "bw_dev" : 116.354139,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 162905,\n'
' "iops_max" : 163665,\n'
' "iops_mean" : '
'163210.428571,\n'
' "iops_stddev" : '
'232.757290,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 13,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 98.485491,\n'
' "20" : 1.498575,\n'
' "50" : 0.010000,\n'
' "100" : 0.013221,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=5227: Wed Oct 26 15:00:32 '
'2022\n'
' read: IOPS=163k, BW=79.7MiB/s '
'(83.5MB/s)(558MiB/7001msec)\n'
' slat (nsec): min=230, '
'max=3617, avg=269.13, '
'stdev=21.63\n'
' clat (usec): min=2, max=111, '
'avg= 5.64, stdev= 1.13\n'
' lat (usec): min=5, max=112, '
'avg= 5.91, stdev= 1.13\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5344], 5.00th=[ '
'5344], 10.00th=[ 5408], 20.00th=[ '
'5408],\n'
' | 30.00th=[ 5472], 40.00th=[ '
'5472], 50.00th=[ 5472], 60.00th=[ '
'5536],\n'
' | 70.00th=[ 5536], 80.00th=[ '
'5600], 90.00th=[ 5664], 95.00th=[ '
'5728],\n'
' | 99.00th=[11456], '
'99.50th=[12480], 99.90th=[16768], '
'99.95th=[17024],\n'
' | 99.99th=[58624]\n'
' bw ( KiB/s): min=81452, '
'max=81832, per=100.00%, '
'avg=81605.14, stdev=116.35, '
'samples=14\n'
' iops : min=162905, '
'max=163665, avg=163210.43, '
'stdev=232.76, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=98.49%, 20=1.50%, 50=0.01%, '
'100=0.01%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=13, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1142152,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=79.7MiB/s (83.5MB/s), '
'79.7MiB/s-79.7MiB/s '
'(83.5MB/s-83.5MB/s), io=558MiB '
'(585MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_21.output'),
'state': {'begin': 1666782142.8388188,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8268139362335205,
'end': 1666782143.6656327,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_21.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_21.state')},
'cmd_210': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790147,\n'
' "timestamp_ms" : '
'1666790147445,\n'
' "time" : "Wed Oct 26 15:15:47 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1479533568,\n'
' "io_kbytes" : 1444857,\n'
' "bw_bytes" : 211331748,\n'
' "bw" : 206378,\n'
' "iops" : 412756.891873,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2889711,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1232,\n'
' "max" : 22802,\n'
' "mean" : 1856.760263,\n'
' "stddev" : '
'376.862601,\n'
' "N" : 2889711\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 140,\n'
' "max" : 125245,\n'
' "mean" : 7527.013691,\n'
' "stddev" : '
'2833.703087,\n'
' "N" : 2889714,\n'
' "percentile" : {\n'
' "1.000000" : 4320,\n'
' "5.000000" : 4896,\n'
' "10.000000" : 5792,\n'
' "20.000000" : 6624,\n'
' "30.000000" : 6688,\n'
' "40.000000" : 7136,\n'
' "50.000000" : 7264,\n'
' "60.000000" : 7328,\n'
' "70.000000" : 7520,\n'
' "80.000000" : 7904,\n'
' "90.000000" : 8512,\n'
' "95.000000" : 9408,\n'
' "99.000000" : '
'20864,\n'
' "99.500000" : '
'25216,\n'
' "99.900000" : '
'27520,\n'
' "99.950000" : '
'34048,\n'
' "99.990000" : 69120\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5701,\n'
' "max" : 127048,\n'
' "mean" : 9383.773646,\n'
' "stddev" : '
'2814.443814,\n'
' "N" : 2889714\n'
' },\n'
' "bw_min" : 205006,\n'
' "bw_max" : 208277,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'206571.076923,\n'
' "bw_dev" : 1108.539088,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 410012,\n'
' "iops_max" : 416554,\n'
' "iops_mean" : '
'413142.153846,\n'
' "iops_stddev" : '
'2216.807195,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 19.342857,\n'
' "sys_cpu" : 76.642857,\n'
' "ctx" : 35152,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 95.738432,\n'
' "20" : 2.739409,\n'
' "50" : 1.478556,\n'
' "100" : 0.041319,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=8503: Wed '
'Oct 26 15:15:47 2022\n'
' read: IOPS=413k, BW=202MiB/s '
'(211MB/s)(1411MiB/7001msec)\n'
' slat (nsec): min=1232, '
'max=22802, avg=1856.76, '
'stdev=376.86\n'
' clat (nsec): min=140, '
'max=125245, avg=7527.01, '
'stdev=2833.70\n'
' lat (usec): min=5, max=127, '
'avg= 9.38, stdev= 2.81\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4320], '
'5.00th=[ 4896], 10.00th=[ 5792], '
'20.00th=[ 6624],\n'
' | 30.00th=[ 6688], '
'40.00th=[ 7136], 50.00th=[ '
'7264], 60.00th=[ 7328],\n'
' | 70.00th=[ 7520], '
'80.00th=[ 7904], 90.00th=[ '
'8512], 95.00th=[ 9408],\n'
' | 99.00th=[20864], '
'99.50th=[25216], '
'99.90th=[27520], '
'99.95th=[34048],\n'
' | 99.99th=[69120]\n'
' bw ( KiB/s): min=205006, '
'max=208277, per=100.00%, '
'avg=206571.08, stdev=1108.54, '
'samples=13\n'
' iops : min=410012, '
'max=416554, avg=413142.15, '
'stdev=2216.81, samples=13\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=95.74%, 20=2.74%, 50=1.48%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=19.34%, '
'sys=76.64%, ctx=35152, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2889711,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=202MiB/s (211MB/s), '
'202MiB/s-202MiB/s '
'(211MB/s-211MB/s), io=1411MiB '
'(1480MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_210.output'),
'state': {'begin': 1666783057.3694043,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.842477560043335,
'end': 1666783058.2118819,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_210.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_210.state')},
'cmd_211': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_211.output'),
'state': {'begin': 1666783059.212763,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7731776237487793,
'end': 1666783059.9859407,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_211.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_211.state')},
'cmd_212': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_212.output'),
'state': {'begin': 1666783059.9877508,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.118003368377686,
'end': 1666783071.1057541,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_212.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_212.state')},
'cmd_213': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790161,\n'
' "timestamp_ms" : '
'1666790161185,\n'
' "time" : "Wed Oct 26 15:16:01 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1513281024,\n'
' "io_kbytes" : 1477813,\n'
' "bw_bytes" : 216152124,\n'
' "bw" : 211086,\n'
' "iops" : 422171.546922,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2955623,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1212,\n'
' "max" : 68398,\n'
' "mean" : 1804.931661,\n'
' "stddev" : '
'383.353388,\n'
' "N" : 2955624\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 130,\n'
' "max" : 121648,\n'
' "mean" : 7361.880692,\n'
' "stddev" : '
'2801.249674,\n'
' "N" : 2955626,\n'
' "percentile" : {\n'
' "1.000000" : 4256,\n'
' "5.000000" : 4832,\n'
' "10.000000" : 5088,\n'
' "20.000000" : 6496,\n'
' "30.000000" : 6560,\n'
' "40.000000" : 6688,\n'
' "50.000000" : 7072,\n'
' "60.000000" : 7200,\n'
' "70.000000" : 7392,\n'
' "80.000000" : 7840,\n'
' "90.000000" : 8384,\n'
' "95.000000" : 9280,\n'
' "99.000000" : '
'20864,\n'
' "99.500000" : '
'24960,\n'
' "99.900000" : '
'27264,\n'
' "99.950000" : '
'33024,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5770,\n'
' "max" : 122910,\n'
' "mean" : 9166.812258,\n'
' "stddev" : '
'2786.079322,\n'
' "N" : 2955626\n'
' },\n'
' "bw_min" : 209124,\n'
' "bw_max" : 212490,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'211123.571429,\n'
' "bw_dev" : 1103.367274,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 418248,\n'
' "iops_max" : 424980,\n'
' "iops_mean" : '
'422247.142857,\n'
' "iops_stddev" : '
'2206.773449,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 19.828571,\n'
' "sys_cpu" : 76.085714,\n'
' "ctx" : 35312,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 95.890816,\n'
' "20" : 2.646379,\n'
' "50" : 1.422881,\n'
' "100" : 0.037623,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=8549: Wed '
'Oct 26 15:16:01 2022\n'
' read: IOPS=422k, BW=206MiB/s '
'(216MB/s)(1443MiB/7001msec)\n'
' slat (nsec): min=1212, '
'max=68398, avg=1804.93, '
'stdev=383.35\n'
' clat (nsec): min=130, '
'max=121648, avg=7361.88, '
'stdev=2801.25\n'
' lat (usec): min=5, max=122, '
'avg= 9.17, stdev= 2.79\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4256], '
'5.00th=[ 4832], 10.00th=[ 5088], '
'20.00th=[ 6496],\n'
' | 30.00th=[ 6560], '
'40.00th=[ 6688], 50.00th=[ '
'7072], 60.00th=[ 7200],\n'
' | 70.00th=[ 7392], '
'80.00th=[ 7840], 90.00th=[ '
'8384], 95.00th=[ 9280],\n'
' | 99.00th=[20864], '
'99.50th=[24960], '
'99.90th=[27264], '
'99.95th=[33024],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=209124, '
'max=212490, per=100.00%, '
'avg=211123.57, stdev=1103.37, '
'samples=14\n'
' iops : min=418248, '
'max=424980, avg=422247.14, '
'stdev=2206.77, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=95.89%, 20=2.65%, '
'50=1.42%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=19.83%, '
'sys=76.09%, ctx=35312, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2955623,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=206MiB/s (216MB/s), '
'206MiB/s-206MiB/s '
'(216MB/s-216MB/s), io=1443MiB '
'(1513MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_213.output'),
'state': {'begin': 1666783071.1096673,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.9034547805786133,
'end': 1666783072.013122,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_213.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_213.state')},
'cmd_214': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_214.output'),
'state': {'begin': 1666783073.0147161,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7947611808776855,
'end': 1666783073.8094773,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_214.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_214.state')},
'cmd_215': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_215.output'),
'state': {'begin': 1666783073.8116484,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.099498748779297,
'end': 1666783084.911147,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_215.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_215.state')},
'cmd_216': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790174,\n'
' "timestamp_ms" : '
'1666790174990,\n'
' "time" : "Wed Oct 26 15:16:14 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1493146112,\n'
' "io_kbytes" : 1458150,\n'
' "bw_bytes" : 213276119,\n'
' "bw" : 208277,\n'
' "iops" : 416554.349379,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2916297,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1222,\n'
' "max" : 23835,\n'
' "mean" : 1835.246559,\n'
' "stddev" : '
'380.983468,\n'
' "N" : 2916298\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 140,\n'
' "max" : 117631,\n'
' "mean" : 7459.801219,\n'
' "stddev" : '
'2783.673537,\n'
' "N" : 2916300,\n'
' "percentile" : {\n'
' "1.000000" : 4256,\n'
' "5.000000" : 4896,\n'
' "10.000000" : 5792,\n'
' "20.000000" : 6560,\n'
' "30.000000" : 6624,\n'
' "40.000000" : 6816,\n'
' "50.000000" : 7200,\n'
' "60.000000" : 7264,\n'
' "70.000000" : 7520,\n'
' "80.000000" : 7904,\n'
' "90.000000" : 8384,\n'
' "95.000000" : 9408,\n'
' "99.000000" : '
'20864,\n'
' "99.500000" : '
'25216,\n'
' "99.900000" : '
'27264,\n'
' "99.950000" : '
'33024,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5740,\n'
' "max" : 119665,\n'
' "mean" : 9295.047879,\n'
' "stddev" : '
'2765.898562,\n'
' "N" : 2916300\n'
' },\n'
' "bw_min" : 206224,\n'
' "bw_max" : 211144,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'208366.357143,\n'
' "bw_dev" : 1490.058678,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 412448,\n'
' "iops_max" : 422288,\n'
' "iops_mean" : '
'416732.642857,\n'
' "iops_stddev" : '
'2980.303175,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 20.971429,\n'
' "sys_cpu" : 75.014286,\n'
' "ctx" : 35185,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 95.807011,\n'
' "20" : 2.722768,\n'
' "50" : 1.432056,\n'
' "100" : 0.036450,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=8596: Wed '
'Oct 26 15:16:14 2022\n'
' read: IOPS=417k, BW=203MiB/s '
'(213MB/s)(1424MiB/7001msec)\n'
' slat (nsec): min=1222, '
'max=23835, avg=1835.25, '
'stdev=380.98\n'
' clat (nsec): min=140, '
'max=117631, avg=7459.80, '
'stdev=2783.67\n'
' lat (usec): min=5, max=119, '
'avg= 9.30, stdev= 2.77\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4256], '
'5.00th=[ 4896], 10.00th=[ 5792], '
'20.00th=[ 6560],\n'
' | 30.00th=[ 6624], '
'40.00th=[ 6816], 50.00th=[ '
'7200], 60.00th=[ 7264],\n'
' | 70.00th=[ 7520], '
'80.00th=[ 7904], 90.00th=[ '
'8384], 95.00th=[ 9408],\n'
' | 99.00th=[20864], '
'99.50th=[25216], '
'99.90th=[27264], '
'99.95th=[33024],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=206224, '
'max=211144, per=100.00%, '
'avg=208366.36, stdev=1490.06, '
'samples=14\n'
' iops : min=412448, '
'max=422288, avg=416732.64, '
'stdev=2980.30, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=95.81%, 20=2.72%, 50=1.43%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=20.97%, '
'sys=75.01%, ctx=35185, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2916297,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=203MiB/s (213MB/s), '
'203MiB/s-203MiB/s '
'(213MB/s-213MB/s), io=1424MiB '
'(1493MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_216.output'),
'state': {'begin': 1666783084.9171014,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8301782608032227,
'end': 1666783085.7472796,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_216.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_216.state')},
'cmd_217': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_217.output'),
'state': {'begin': 1666783087.8358128,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8178656101226807,
'end': 1666783088.6536784,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_217.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_217.state')},
'cmd_218': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_218.output'),
'state': {'begin': 1666783088.6558793,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.464780807495117,
'end': 1666783100.12066,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_218.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_218.state')},
'cmd_219': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790190,\n'
' "timestamp_ms" : '
'1666790190135,\n'
' "time" : "Wed Oct 26 15:16:30 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1345384448,\n'
' "io_kbytes" : 1313852,\n'
' "bw_bytes" : 192170325,\n'
' "bw" : 187666,\n'
' "iops" : 375332.238252,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2627701,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 651,\n'
' "max" : 13565,\n'
' "mean" : 938.106651,\n'
' "stddev" : 69.869104,\n'
' "N" : 2627701\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1402,\n'
' "max" : 108673,\n'
' "mean" : 9369.322797,\n'
' "stddev" : '
'2206.289705,\n'
' "N" : 2627704,\n'
' "percentile" : {\n'
' "1.000000" : 7712,\n'
' "5.000000" : 8256,\n'
' "10.000000" : 8512,\n'
' "20.000000" : 8640,\n'
' "30.000000" : 8896,\n'
' "40.000000" : 8896,\n'
' "50.000000" : 9024,\n'
' "60.000000" : 9152,\n'
' "70.000000" : 9280,\n'
' "80.000000" : 9408,\n'
' "90.000000" : 9664,\n'
' "95.000000" : '
'10688,\n'
' "99.000000" : '
'20096,\n'
' "99.500000" : '
'24192,\n'
' "99.900000" : '
'29056,\n'
' "99.950000" : '
'34048,\n'
' "99.990000" : 67072\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7174,\n'
' "max" : 109625,\n'
' "mean" : '
'10307.429430,\n'
' "stddev" : '
'2207.091785,\n'
' "N" : 2627704\n'
' },\n'
' "bw_min" : 187307,\n'
' "bw_max" : 188201,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'187705.857143,\n'
' "bw_dev" : 270.564327,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 374614,\n'
' "iops_max" : 376402,\n'
' "iops_mean" : '
'375411.785714,\n'
' "iops_stddev" : '
'541.060523,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 12,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 3.338127,\n'
' "4" : 96.661683,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 93.003199,\n'
' "20" : 5.969895,\n'
' "50" : 0.989154,\n'
' "100" : 0.037714,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=8655: Wed Oct 26 15:16:30 '
'2022\n'
' read: IOPS=375k, BW=183MiB/s '
'(192MB/s)(1283MiB/7001msec)\n'
' slat (nsec): min=651, '
'max=13565, avg=938.11, '
'stdev=69.87\n'
' clat (nsec): min=1402, '
'max=108673, avg=9369.32, '
'stdev=2206.29\n'
' lat (usec): min=7, max=109, '
'avg=10.31, stdev= 2.21\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7712], '
'5.00th=[ 8256], 10.00th=[ 8512], '
'20.00th=[ 8640],\n'
' | 30.00th=[ 8896], '
'40.00th=[ 8896], 50.00th=[ '
'9024], 60.00th=[ 9152],\n'
' | 70.00th=[ 9280], '
'80.00th=[ 9408], 90.00th=[ '
'9664], 95.00th=[10688],\n'
' | 99.00th=[20096], '
'99.50th=[24192], '
'99.90th=[29056], '
'99.95th=[34048],\n'
' | 99.99th=[67072]\n'
' bw ( KiB/s): min=187307, '
'max=188201, per=100.00%, '
'avg=187705.86, stdev=270.56, '
'samples=14\n'
' iops : min=374614, '
'max=376402, avg=375411.79, '
'stdev=541.06, samples=14\n'
' lat (usec) : 2=0.01%, '
'10=93.00%, 20=5.97%, 50=0.99%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=12, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=3.3%, '
'4=96.7%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2627701,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=183MiB/s (192MB/s), '
'183MiB/s-183MiB/s '
'(192MB/s-192MB/s), io=1283MiB '
'(1345MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_219.output'),
'state': {'begin': 1666783100.123212,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8393714427947998,
'end': 1666783100.9625835,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_219.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_219.state')},
'cmd_22': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_22.output'),
'state': {'begin': 1666782145.8041782,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8044922351837158,
'end': 1666782146.6086705,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_22.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_22.state')},
'cmd_220': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_220.output'),
'state': {'begin': 1666783103.0728974,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8001556396484375,
'end': 1666783103.873053,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_220.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_220.state')},
'cmd_221': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_221.output'),
'state': {'begin': 1666783103.875193,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.461009740829468,
'end': 1666783115.3362029,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_221.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_221.state')},
'cmd_222': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790205,\n'
' "timestamp_ms" : '
'1666790205340,\n'
' "time" : "Wed Oct 26 15:16:45 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1416632832,\n'
' "io_kbytes" : 1383430,\n'
' "bw_bytes" : 202347212,\n'
' "bw" : 197604,\n'
' "iops" : 395208.970147,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2766858,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 171,\n'
' "max" : 16481,\n'
' "mean" : 269.504734,\n'
' "stddev" : 26.227924,\n'
' "N" : 2766858\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 380,\n'
' "max" : 121328,\n'
' "mean" : 9537.189593,\n'
' "stddev" : '
'2328.868613,\n'
' "N" : 2766861,\n'
' "percentile" : {\n'
' "1.000000" : 6432,\n'
' "5.000000" : 7392,\n'
' "10.000000" : 8032,\n'
' "20.000000" : 8768,\n'
' "30.000000" : 9024,\n'
' "40.000000" : 9152,\n'
' "50.000000" : 9280,\n'
' "60.000000" : 9408,\n'
' "70.000000" : 9536,\n'
' "80.000000" : 9664,\n'
' "90.000000" : '
'10304,\n'
' "95.000000" : '
'11840,\n'
' "99.000000" : '
'20096,\n'
' "99.500000" : '
'21888,\n'
' "99.900000" : '
'27264,\n'
' "99.950000" : '
'33024,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5581,\n'
' "max" : 121598,\n'
' "mean" : 9806.694310,\n'
' "stddev" : '
'2330.569751,\n'
' "N" : 2766861\n'
' },\n'
' "bw_min" : 197016,\n'
' "bw_max" : 198564,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'197684.571429,\n'
' "bw_dev" : 480.434213,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 394032,\n'
' "iops_max" : 397128,\n'
' "iops_mean" : '
'395369.142857,\n'
' "iops_stddev" : '
'960.868426,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 14,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 12.759744,\n'
' "4" : 87.237184,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 87.316841,\n'
' "20" : 11.658495,\n'
' "50" : 0.982631,\n'
' "100" : 0.041672,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=8715: Wed Oct 26 15:16:45 '
'2022\n'
' read: IOPS=395k, BW=193MiB/s '
'(202MB/s)(1351MiB/7001msec)\n'
' slat (nsec): min=171, '
'max=16481, avg=269.50, '
'stdev=26.23\n'
' clat (nsec): min=380, '
'max=121328, avg=9537.19, '
'stdev=2328.87\n'
' lat (usec): min=5, max=121, '
'avg= 9.81, stdev= 2.33\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6432], '
'5.00th=[ 7392], 10.00th=[ 8032], '
'20.00th=[ 8768],\n'
' | 30.00th=[ 9024], '
'40.00th=[ 9152], 50.00th=[ '
'9280], 60.00th=[ 9408],\n'
' | 70.00th=[ 9536], '
'80.00th=[ 9664], '
'90.00th=[10304], '
'95.00th=[11840],\n'
' | 99.00th=[20096], '
'99.50th=[21888], '
'99.90th=[27264], '
'99.95th=[33024],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=197016, '
'max=198564, per=100.00%, '
'avg=197684.57, stdev=480.43, '
'samples=14\n'
' iops : min=394032, '
'max=397128, avg=395369.14, '
'stdev=960.87, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 10=87.32%, '
'20=11.66%, 50=0.98%, 100=0.04%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=14, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=12.8%, 4=87.2%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2766858,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=193MiB/s (202MB/s), '
'193MiB/s-193MiB/s '
'(202MB/s-202MB/s), io=1351MiB '
'(1417MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_222.output'),
'state': {'begin': 1666783115.3390436,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7917187213897705,
'end': 1666783116.1307623,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_222.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_222.state')},
'cmd_223': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_223.output'),
'state': {'begin': 1666783118.236984,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8149440288543701,
'end': 1666783119.051928,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_223.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_223.state')},
'cmd_224': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_224.output'),
'state': {'begin': 1666783119.0544753,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.496715068817139,
'end': 1666783130.5511904,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_224.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_224.state')},
'cmd_225': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790220,\n'
' "timestamp_ms" : '
'1666790220567,\n'
' "time" : "Wed Oct 26 15:17:00 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1625583104,\n'
' "io_kbytes" : 1587483,\n'
' "bw_bytes" : 232192987,\n'
' "bw" : 226750,\n'
' "iops" : 453501.499786,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3174964,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 200,\n'
' "max" : 19697,\n'
' "mean" : 283.869127,\n'
' "stddev" : 75.463763,\n'
' "N" : 3174964\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 501,\n'
' "max" : 111770,\n'
' "mean" : 8226.494525,\n'
' "stddev" : '
'2228.476383,\n'
' "N" : 3174967,\n'
' "percentile" : {\n'
' "1.000000" : 6368,\n'
' "5.000000" : 6752,\n'
' "10.000000" : 7008,\n'
' "20.000000" : 7456,\n'
' "30.000000" : 7584,\n'
' "40.000000" : 7648,\n'
' "50.000000" : 7776,\n'
' "60.000000" : 7904,\n'
' "70.000000" : 8096,\n'
' "80.000000" : 8512,\n'
' "90.000000" : 9280,\n'
' "95.000000" : '
'10176,\n'
' "99.000000" : '
'18816,\n'
' "99.500000" : '
'21376,\n'
' "99.900000" : '
'27520,\n'
' "99.950000" : '
'33536,\n'
' "99.990000" : 67072\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5861,\n'
' "max" : 112050,\n'
' "mean" : 8510.363694,\n'
' "stddev" : '
'2229.327804,\n'
' "N" : 3174967\n'
' },\n'
' "bw_min" : 226564,\n'
' "bw_max" : 227133,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'226841.785714,\n'
' "bw_dev" : 143.797496,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 453128,\n'
' "iops_max" : 454266,\n'
' "iops_mean" : '
'453683.642857,\n'
' "iops_stddev" : '
'287.590416,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 1.998952,\n'
' "4" : 97.991001,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 94.702460,\n'
' "20" : 4.585154,\n'
' "50" : 0.674307,\n'
' "100" : 0.037197,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=8773: Wed Oct 26 15:17:00 '
'2022\n'
' read: IOPS=454k, BW=221MiB/s '
'(232MB/s)(1550MiB/7001msec)\n'
' slat (nsec): min=200, '
'max=19697, avg=283.87, '
'stdev=75.46\n'
' clat (nsec): min=501, '
'max=111770, avg=8226.49, '
'stdev=2228.48\n'
' lat (usec): min=5, max=112, '
'avg= 8.51, stdev= 2.23\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6368], '
'5.00th=[ 6752], 10.00th=[ 7008], '
'20.00th=[ 7456],\n'
' | 30.00th=[ 7584], '
'40.00th=[ 7648], 50.00th=[ '
'7776], 60.00th=[ 7904],\n'
' | 70.00th=[ 8096], '
'80.00th=[ 8512], 90.00th=[ '
'9280], 95.00th=[10176],\n'
' | 99.00th=[18816], '
'99.50th=[21376], '
'99.90th=[27520], '
'99.95th=[33536],\n'
' | 99.99th=[67072]\n'
' bw ( KiB/s): min=226564, '
'max=227133, per=100.00%, '
'avg=226841.79, stdev=143.80, '
'samples=14\n'
' iops : min=453128, '
'max=454266, avg=453683.64, '
'stdev=287.59, samples=14\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 10=94.70%, '
'20=4.59%, 50=0.67%, 100=0.04%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=2.0%, '
'4=98.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3174964,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=221MiB/s (232MB/s), '
'221MiB/s-221MiB/s '
'(232MB/s-232MB/s), io=1550MiB '
'(1626MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_225.output'),
'state': {'begin': 1666783130.5537598,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8054802417755127,
'end': 1666783131.35924,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_225.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_225.state')},
'cmd_226': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_226.output'),
'state': {'begin': 1666783132.3794498,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8866932392120361,
'end': 1666783133.266143,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_226.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_226.state')},
'cmd_227': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_227.output'),
'state': {'begin': 1666783133.2681544,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.052165508270264,
'end': 1666783144.32032,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_227.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_227.state')},
'cmd_228': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790234,\n'
' "timestamp_ms" : '
'1666790234399,\n'
' "time" : "Wed Oct 26 15:17:14 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1530035200,\n'
' "io_kbytes" : 1494175,\n'
' "bw_bytes" : 218545236,\n'
' "bw" : 213423,\n'
' "iops" : 426845.593487,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2988346,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1313,\n'
' "max" : 23915,\n'
' "mean" : 1844.950892,\n'
' "stddev" : '
'353.846891,\n'
' "N" : 2988347\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 110,\n'
' "max" : 126256,\n'
' "mean" : 7246.118301,\n'
' "stddev" : '
'2909.869603,\n'
' "N" : 2988349,\n'
' "percentile" : {\n'
' "1.000000" : 4192,\n'
' "5.000000" : 4320,\n'
' "10.000000" : 4768,\n'
' "20.000000" : 6240,\n'
' "30.000000" : 6496,\n'
' "40.000000" : 6560,\n'
' "50.000000" : 6752,\n'
' "60.000000" : 7072,\n'
' "70.000000" : 7520,\n'
' "80.000000" : 8256,\n'
' "90.000000" : 8768,\n'
' "95.000000" : 9408,\n'
' "99.000000" : '
'20608,\n'
' "99.500000" : '
'24960,\n'
' "99.900000" : '
'27008,\n'
' "99.950000" : '
'33024,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5380,\n'
' "max" : 127619,\n'
' "mean" : 9091.069296,\n'
' "stddev" : '
'2889.364715,\n'
' "N" : 2988349\n'
' },\n'
' "bw_min" : 212813,\n'
' "bw_max" : 214359,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'213525.857143,\n'
' "bw_dev" : 374.802688,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 425627,\n'
' "iops_max" : 428718,\n'
' "iops_mean" : '
'427051.928571,\n'
' "iops_stddev" : '
'749.479559,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.014286,\n'
' "sys_cpu" : 78.814286,\n'
' "ctx" : 36626,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 95.847569,\n'
' "20" : 2.632359,\n'
' "50" : 1.478845,\n'
' "100" : 0.039487,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 4302124,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 23026,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 23026,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=8823: Wed '
'Oct 26 15:17:14 2022\n'
' read: IOPS=427k, BW=208MiB/s '
'(219MB/s)(1459MiB/7001msec)\n'
' slat (nsec): min=1313, '
'max=23915, avg=1844.95, '
'stdev=353.85\n'
' clat (nsec): min=110, '
'max=126256, avg=7246.12, '
'stdev=2909.87\n'
' lat (usec): min=5, max=127, '
'avg= 9.09, stdev= 2.89\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4192], '
'5.00th=[ 4320], 10.00th=[ 4768], '
'20.00th=[ 6240],\n'
' | 30.00th=[ 6496], '
'40.00th=[ 6560], 50.00th=[ '
'6752], 60.00th=[ 7072],\n'
' | 70.00th=[ 7520], '
'80.00th=[ 8256], 90.00th=[ '
'8768], 95.00th=[ 9408],\n'
' | 99.00th=[20608], '
'99.50th=[24960], '
'99.90th=[27008], '
'99.95th=[33024],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=212813, '
'max=214359, per=100.00%, '
'avg=213525.86, stdev=374.80, '
'samples=14\n'
' iops : min=425627, '
'max=428718, avg=427051.93, '
'stdev=749.48, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=95.85%, 20=2.63%, '
'50=1.48%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=17.01%, '
'sys=78.81%, ctx=36626, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2988346,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=208MiB/s (219MB/s), '
'208MiB/s-208MiB/s '
'(219MB/s-219MB/s), io=1459MiB '
'(1530MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=4302124/0, '
'merge=0/0, ticks=23026/0, '
'in_queue=23026, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_228.output'),
'state': {'begin': 1666783144.3229122,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8065760135650635,
'end': 1666783145.1294882,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_228.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_228.state')},
'cmd_229': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_229.output'),
'state': {'begin': 1666783146.164021,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.798626184463501,
'end': 1666783146.9626472,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_229.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_229.state')},
'cmd_23': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_23.output'),
'state': {'begin': 1666782146.6109998,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.522717714309692,
'end': 1666782158.1337175,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_23.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_23.state')},
'cmd_230': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_230.output'),
'state': {'begin': 1666783146.9648237,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.045536041259766,
'end': 1666783158.0103598,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_230.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_230.state')},
'cmd_231': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790248,\n'
' "timestamp_ms" : '
'1666790248087,\n'
' "time" : "Wed Oct 26 15:17:28 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1519099904,\n'
' "io_kbytes" : 1483496,\n'
' "bw_bytes" : 216983274,\n'
' "bw" : 211897,\n'
' "iops" : 423794.886445,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2966988,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1322,\n'
' "max" : 22562,\n'
' "mean" : 1861.858082,\n'
' "stddev" : '
'352.891983,\n'
' "N" : 2966989\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 100,\n'
' "max" : 122830,\n'
' "mean" : 7297.526278,\n'
' "stddev" : '
'2887.510987,\n'
' "N" : 2966991,\n'
' "percentile" : {\n'
' "1.000000" : 4192,\n'
' "5.000000" : 4320,\n'
' "10.000000" : 4832,\n'
' "20.000000" : 6368,\n'
' "30.000000" : 6560,\n'
' "40.000000" : 6624,\n'
' "50.000000" : 6880,\n'
' "60.000000" : 7200,\n'
' "70.000000" : 7520,\n'
' "80.000000" : 8256,\n'
' "90.000000" : 8768,\n'
' "95.000000" : 9408,\n'
' "99.000000" : '
'20608,\n'
' "99.500000" : '
'24960,\n'
' "99.900000" : '
'27008,\n'
' "99.950000" : '
'33024,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5731,\n'
' "max" : 124212,\n'
' "mean" : 9159.384584,\n'
' "stddev" : '
'2868.348094,\n'
' "N" : 2966991\n'
' },\n'
' "bw_min" : 211271,\n'
' "bw_max" : 212946,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'212027.500000,\n'
' "bw_dev" : 425.605225,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 422542,\n'
' "iops_max" : 425892,\n'
' "iops_mean" : '
'424055.357143,\n'
' "iops_stddev" : '
'851.333308,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.200000,\n'
' "sys_cpu" : 78.642857,\n'
' "ctx" : 35976,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 95.813296,\n'
' "20" : 2.761386,\n'
' "50" : 1.384872,\n'
' "100" : 0.039299,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 4273785,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 22940,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 22940,\n'
' "util" : 99.098564\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=8869: Wed '
'Oct 26 15:17:28 2022\n'
' read: IOPS=424k, BW=207MiB/s '
'(217MB/s)(1449MiB/7001msec)\n'
' slat (nsec): min=1322, '
'max=22562, avg=1861.86, '
'stdev=352.89\n'
' clat (nsec): min=100, '
'max=122830, avg=7297.53, '
'stdev=2887.51\n'
' lat (usec): min=5, max=124, '
'avg= 9.16, stdev= 2.87\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4192], '
'5.00th=[ 4320], 10.00th=[ 4832], '
'20.00th=[ 6368],\n'
' | 30.00th=[ 6560], '
'40.00th=[ 6624], 50.00th=[ '
'6880], 60.00th=[ 7200],\n'
' | 70.00th=[ 7520], '
'80.00th=[ 8256], 90.00th=[ '
'8768], 95.00th=[ 9408],\n'
' | 99.00th=[20608], '
'99.50th=[24960], '
'99.90th=[27008], '
'99.95th=[33024],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=211271, '
'max=212946, per=100.00%, '
'avg=212027.50, stdev=425.61, '
'samples=14\n'
' iops : min=422542, '
'max=425892, avg=424055.36, '
'stdev=851.33, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=95.81%, 20=2.76%, '
'50=1.38%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=17.20%, '
'sys=78.64%, ctx=35976, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2966988,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=207MiB/s (217MB/s), '
'207MiB/s-207MiB/s '
'(217MB/s-217MB/s), io=1449MiB '
'(1519MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=4273785/0, '
'merge=0/0, ticks=22940/0, '
'in_queue=22940, util=99.10%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_231.output'),
'state': {'begin': 1666783158.0133226,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8554275035858154,
'end': 1666783158.86875,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_231.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_231.state')},
'cmd_232': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_232.output'),
'state': {'begin': 1666783159.9105759,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7936725616455078,
'end': 1666783160.7042484,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_232.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_232.state')},
'cmd_233': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_233.output'),
'state': {'begin': 1666783160.7061718,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.07721996307373,
'end': 1666783171.7833917,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_233.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_233.state')},
'cmd_234': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790261,\n'
' "timestamp_ms" : '
'1666790261828,\n'
' "time" : "Wed Oct 26 15:17:41 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1502555648,\n'
' "io_kbytes" : 1467339,\n'
' "bw_bytes" : 214620146,\n'
' "bw" : 209589,\n'
' "iops" : 419179.402942,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2934675,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1333,\n'
' "max" : 17603,\n'
' "mean" : 1865.526482,\n'
' "stddev" : '
'355.622282,\n'
' "N" : 2934676\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 100,\n'
' "max" : 118191,\n'
' "mean" : 7382.369814,\n'
' "stddev" : '
'2876.847571,\n'
' "N" : 2934678,\n'
' "percentile" : {\n'
' "1.000000" : 4256,\n'
' "5.000000" : 4320,\n'
' "10.000000" : 4896,\n'
' "20.000000" : 6496,\n'
' "30.000000" : 6624,\n'
' "40.000000" : 6752,\n'
' "50.000000" : 7136,\n'
' "60.000000" : 7264,\n'
' "70.000000" : 7648,\n'
' "80.000000" : 8256,\n'
' "90.000000" : 8640,\n'
' "95.000000" : 9408,\n'
' "99.000000" : '
'20864,\n'
' "99.500000" : '
'25216,\n'
' "99.900000" : '
'27008,\n'
' "99.950000" : '
'33024,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5760,\n'
' "max" : 119564,\n'
' "mean" : 9247.896274,\n'
' "stddev" : '
'2865.318682,\n'
' "N" : 2934678\n'
' },\n'
' "bw_min" : 208949,\n'
' "bw_max" : 210632,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'209732.785714,\n'
' "bw_dev" : 576.971694,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 417898,\n'
' "iops_max" : 421264,\n'
' "iops_mean" : '
'419465.785714,\n'
' "iops_stddev" : '
'1153.870889,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.242857,\n'
' "sys_cpu" : 78.757143,\n'
' "ctx" : 32783,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 95.971786,\n'
' "20" : 2.527912,\n'
' "50" : 1.462308,\n'
' "100" : 0.035949,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 4230620,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 22672,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 22672,\n'
' "util" : 99.098564\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=8917: Wed '
'Oct 26 15:17:41 2022\n'
' read: IOPS=419k, BW=205MiB/s '
'(215MB/s)(1433MiB/7001msec)\n'
' slat (nsec): min=1333, '
'max=17603, avg=1865.53, '
'stdev=355.62\n'
' clat (nsec): min=100, '
'max=118191, avg=7382.37, '
'stdev=2876.85\n'
' lat (usec): min=5, max=119, '
'avg= 9.25, stdev= 2.87\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4256], '
'5.00th=[ 4320], 10.00th=[ 4896], '
'20.00th=[ 6496],\n'
' | 30.00th=[ 6624], '
'40.00th=[ 6752], 50.00th=[ '
'7136], 60.00th=[ 7264],\n'
' | 70.00th=[ 7648], '
'80.00th=[ 8256], 90.00th=[ '
'8640], 95.00th=[ 9408],\n'
' | 99.00th=[20864], '
'99.50th=[25216], '
'99.90th=[27008], '
'99.95th=[33024],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=208949, '
'max=210632, per=100.00%, '
'avg=209732.79, stdev=576.97, '
'samples=14\n'
' iops : min=417898, '
'max=421264, avg=419465.79, '
'stdev=1153.87, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=95.97%, 20=2.53%, 50=1.46%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=17.24%, '
'sys=78.76%, ctx=32783, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2934675,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=205MiB/s (215MB/s), '
'205MiB/s-205MiB/s '
'(215MB/s-215MB/s), io=1433MiB '
'(1503MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=4230620/0, '
'merge=0/0, ticks=22672/0, '
'in_queue=22672, util=99.10%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_234.output'),
'state': {'begin': 1666783171.7861347,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7979645729064941,
'end': 1666783172.5840993,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_234.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_234.state')},
'cmd_235': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_235.output'),
'state': {'begin': 1666783174.6949475,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.7767727375030518,
'end': 1666783175.4717202,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_235.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_235.state')},
'cmd_236': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_236.output'),
'state': {'begin': 1666783175.4740148,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.314560413360596,
'end': 1666783186.7885752,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_236.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_236.state')},
'cmd_237': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790276,\n'
' "timestamp_ms" : '
'1666790276804,\n'
' "time" : "Wed Oct 26 15:17:56 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1205324288,\n'
' "io_kbytes" : 1177074,\n'
' "bw_bytes" : 172164589,\n'
' "bw" : 168129,\n'
' "iops" : 336258.534495,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2354146,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 17002,\n'
' "mean" : 179.156082,\n'
' "stddev" : '
'149.002890,\n'
' "N" : 2354146\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5971,\n'
' "max" : 115316,\n'
' "mean" : '
'11061.462771,\n'
' "stddev" : '
'3262.876327,\n'
' "N" : 2354147,\n'
' "percentile" : {\n'
' "1.000000" : 6496,\n'
' "5.000000" : 7328,\n'
' "10.000000" : 7968,\n'
' "20.000000" : 8384,\n'
' "30.000000" : 8768,\n'
' "40.000000" : 9152,\n'
' "50.000000" : '
'10432,\n'
' "60.000000" : '
'12096,\n'
' "70.000000" : '
'12736,\n'
' "80.000000" : '
'13248,\n'
' "90.000000" : '
'13760,\n'
' "95.000000" : '
'17280,\n'
' "99.000000" : '
'21376,\n'
' "99.500000" : '
'24192,\n'
' "99.900000" : '
'30336,\n'
' "99.950000" : '
'35072,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6171,\n'
' "max" : 115496,\n'
' "mean" : '
'11240.618823,\n'
' "stddev" : '
'3234.373994,\n'
' "N" : 2354147\n'
' },\n'
' "bw_min" : 166257,\n'
' "bw_max" : 170589,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'168231.142857,\n'
' "bw_dev" : 1493.829661,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 332515,\n'
' "iops_max" : 341178,\n'
' "iops_mean" : '
'336462.428571,\n'
' "iops_stddev" : '
'2987.544238,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 41.185714,\n'
' "sys_cpu" : 58.785714,\n'
' "ctx" : 11,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 6.693255,\n'
' "2" : 43.215119,\n'
' "4" : 50.091626,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 47.577253,\n'
' "20" : 51.221589,\n'
' "50" : 1.163182,\n'
' "100" : 0.037848,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=8974: Wed '
'Oct 26 15:17:56 2022\n'
' read: IOPS=336k, BW=164MiB/s '
'(172MB/s)(1149MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=17002, avg=179.16, '
'stdev=149.00\n'
' clat (usec): min=5, max=115, '
'avg=11.06, stdev= 3.26\n'
' lat (usec): min=6, max=115, '
'avg=11.24, stdev= 3.23\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6496], '
'5.00th=[ 7328], 10.00th=[ 7968], '
'20.00th=[ 8384],\n'
' | 30.00th=[ 8768], '
'40.00th=[ 9152], '
'50.00th=[10432], '
'60.00th=[12096],\n'
' | 70.00th=[12736], '
'80.00th=[13248], '
'90.00th=[13760], '
'95.00th=[17280],\n'
' | 99.00th=[21376], '
'99.50th=[24192], '
'99.90th=[30336], '
'99.95th=[35072],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=166257, '
'max=170589, per=100.00%, '
'avg=168231.14, stdev=1493.83, '
'samples=14\n'
' iops : min=332515, '
'max=341178, avg=336462.43, '
'stdev=2987.54, samples=14\n'
' lat (usec) : 10=47.58%, '
'20=51.22%, 50=1.16%, 100=0.04%, '
'250=0.01%\n'
' cpu : usr=41.19%, '
'sys=58.79%, ctx=11, majf=0, '
'minf=0\n'
' IO depths : 1=6.7%, '
'2=43.2%, 4=50.1%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2354146,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=164MiB/s (172MB/s), '
'164MiB/s-164MiB/s '
'(172MB/s-172MB/s), io=1149MiB '
'(1205MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_237.output'),
'state': {'begin': 1666783186.7954512,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.816450834274292,
'end': 1666783187.611902,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_237.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_237.state')},
'cmd_238': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_238.output'),
'state': {'begin': 1666783189.7250276,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7946679592132568,
'end': 1666783190.5196955,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_238.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_238.state')},
'cmd_239': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_239.output'),
'state': {'begin': 1666783190.5224497,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.44857120513916,
'end': 1666783201.971021,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_239.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_239.state')},
'cmd_24': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789248,\n'
' "timestamp_ms" : '
'1666789248165,\n'
' "time" : "Wed Oct 26 15:00:48 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 407997952,\n'
' "io_kbytes" : 398435,\n'
' "bw_bytes" : 58277096,\n'
' "bw" : 56911,\n'
' "iops" : 113822.453935,\n'
' "runtime" : 7001,\n'
' "total_ios" : 796871,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 711,\n'
' "max" : 20899,\n'
' "mean" : 921.766521,\n'
' "stddev" : 69.333618,\n'
' "N" : 796871\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5120,\n'
' "max" : 80270,\n'
' "mean" : 7528.237944,\n'
' "stddev" : '
'1123.122707,\n'
' "N" : 796871,\n'
' "percentile" : {\n'
' "1.000000" : 6880,\n'
' "5.000000" : 7008,\n'
' "10.000000" : 7072,\n'
' "20.000000" : 7200,\n'
' "30.000000" : 7264,\n'
' "40.000000" : 7328,\n'
' "50.000000" : 7392,\n'
' "60.000000" : 7520,\n'
' "70.000000" : 7584,\n'
' "80.000000" : 7648,\n'
' "90.000000" : 7776,\n'
' "95.000000" : 7904,\n'
' "99.000000" : 11712,\n'
' "99.500000" : 14400,\n'
' "99.900000" : 16512,\n'
' "99.950000" : 17280,\n'
' "99.990000" : 60672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7153,\n'
' "max" : 81202,\n'
' "mean" : 8450.004465,\n'
' "stddev" : '
'1126.519340,\n'
' "N" : 796871\n'
' },\n'
' "bw_min" : 56840,\n'
' "bw_max" : 57028,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'56925.714286,\n'
' "bw_dev" : 58.529977,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 113680,\n'
' "iops_max" : 114056,\n'
' "iops_mean" : '
'113851.500000,\n'
' "iops_stddev" : '
'117.153909,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 11,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 98.200336,\n'
' "20" : 1.771805,\n'
' "50" : 0.010000,\n'
' "100" : 0.018322,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=5285: Wed Oct 26 15:00:48 '
'2022\n'
' read: IOPS=114k, BW=55.6MiB/s '
'(58.3MB/s)(389MiB/7001msec)\n'
' slat (nsec): min=711, '
'max=20899, avg=921.77, '
'stdev=69.33\n'
' clat (nsec): min=5120, '
'max=80270, avg=7528.24, '
'stdev=1123.12\n'
' lat (nsec): min=7153, '
'max=81202, avg=8450.00, '
'stdev=1126.52\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6880], 5.00th=[ '
'7008], 10.00th=[ 7072], 20.00th=[ '
'7200],\n'
' | 30.00th=[ 7264], 40.00th=[ '
'7328], 50.00th=[ 7392], 60.00th=[ '
'7520],\n'
' | 70.00th=[ 7584], 80.00th=[ '
'7648], 90.00th=[ 7776], 95.00th=[ '
'7904],\n'
' | 99.00th=[11712], '
'99.50th=[14400], 99.90th=[16512], '
'99.95th=[17280],\n'
' | 99.99th=[60672]\n'
' bw ( KiB/s): min=56840, '
'max=57028, per=100.00%, '
'avg=56925.71, stdev=58.53, '
'samples=14\n'
' iops : min=113680, '
'max=114056, avg=113851.50, '
'stdev=117.15, samples=14\n'
' lat (usec) : 10=98.20%, '
'20=1.77%, 50=0.01%, 100=0.02%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=11, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=796871,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=55.6MiB/s (58.3MB/s), '
'55.6MiB/s-55.6MiB/s '
'(58.3MB/s-58.3MB/s), io=389MiB '
'(408MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_24.output'),
'state': {'begin': 1666782158.1400828,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7942419052124023,
'end': 1666782158.9343247,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_24.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_24.state')},
'cmd_240': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790291,\n'
' "timestamp_ms" : '
'1666790291984,\n'
' "time" : "Wed Oct 26 15:18:11 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1173137408,\n'
' "io_kbytes" : 1145642,\n'
' "bw_bytes" : 167591058,\n'
' "bw" : 163663,\n'
' "iops" : 327325.857143,\n'
' "runtime" : 7000,\n'
' "total_ios" : 2291281,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 120,\n'
' "max" : 15419,\n'
' "mean" : 174.303148,\n'
' "stddev" : '
'129.262014,\n'
' "N" : 2291281\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 6031,\n'
' "max" : 97032,\n'
' "mean" : '
'11358.835785,\n'
' "stddev" : '
'3221.532516,\n'
' "N" : 2291284,\n'
' "percentile" : {\n'
' "1.000000" : 6496,\n'
' "5.000000" : 7392,\n'
' "10.000000" : 8032,\n'
' "20.000000" : 8512,\n'
' "30.000000" : 8896,\n'
' "40.000000" : 9664,\n'
' "50.000000" : '
'11712,\n'
' "60.000000" : '
'12608,\n'
' "70.000000" : '
'12992,\n'
' "80.000000" : '
'13376,\n'
' "90.000000" : '
'13760,\n'
' "95.000000" : '
'17536,\n'
' "99.000000" : '
'21376,\n'
' "99.500000" : '
'23936,\n'
' "99.900000" : '
'29568,\n'
' "99.950000" : '
'35072,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6202,\n'
' "max" : 97222,\n'
' "mean" : '
'11533.138909,\n'
' "stddev" : '
'3199.841005,\n'
' "N" : 2291284\n'
' },\n'
' "bw_min" : 161805,\n'
' "bw_max" : 165426,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'163730.928571,\n'
' "bw_dev" : 1280.287135,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 323611,\n'
' "iops_max" : 330852,\n'
' "iops_mean" : '
'327461.928571,\n'
' "iops_stddev" : '
'2560.458567,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 42.842857,\n'
' "sys_cpu" : 57.128571,\n'
' "ctx" : 16,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 9.001777,\n'
' "2" : 44.955944,\n'
' "4" : 46.042279,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 41.212099,\n'
' "20" : 57.586259,\n'
' "50" : 1.166116,\n'
' "100" : 0.035657,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=9031: Wed '
'Oct 26 15:18:11 2022\n'
' read: IOPS=327k, BW=160MiB/s '
'(168MB/s)(1119MiB/7000msec)\n'
' slat (nsec): min=120, '
'max=15419, avg=174.30, '
'stdev=129.26\n'
' clat (nsec): min=6031, '
'max=97032, avg=11358.84, '
'stdev=3221.53\n'
' lat (nsec): min=6202, '
'max=97222, avg=11533.14, '
'stdev=3199.84\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6496], '
'5.00th=[ 7392], 10.00th=[ 8032], '
'20.00th=[ 8512],\n'
' | 30.00th=[ 8896], '
'40.00th=[ 9664], '
'50.00th=[11712], '
'60.00th=[12608],\n'
' | 70.00th=[12992], '
'80.00th=[13376], '
'90.00th=[13760], '
'95.00th=[17536],\n'
' | 99.00th=[21376], '
'99.50th=[23936], '
'99.90th=[29568], '
'99.95th=[35072],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=161805, '
'max=165426, per=100.00%, '
'avg=163730.93, stdev=1280.29, '
'samples=14\n'
' iops : min=323611, '
'max=330852, avg=327461.93, '
'stdev=2560.46, samples=14\n'
' lat (usec) : 10=41.21%, '
'20=57.59%, 50=1.17%, 100=0.04%\n'
' cpu : usr=42.84%, '
'sys=57.13%, ctx=16, majf=0, '
'minf=0\n'
' IO depths : 1=9.0%, '
'2=45.0%, 4=46.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2291281,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=160MiB/s (168MB/s), '
'160MiB/s-160MiB/s '
'(168MB/s-168MB/s), io=1119MiB '
'(1173MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_240.output'),
'state': {'begin': 1666783201.9737847,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.8595907688140869,
'end': 1666783202.8333755,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_240.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_240.state')},
'cmd_241': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_241.output'),
'state': {'begin': 1666783204.9390643,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.7746074199676514,
'end': 1666783205.7136717,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_241.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_241.state')},
'cmd_242': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_242.output'),
'state': {'begin': 1666783205.7160082,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.491530179977417,
'end': 1666783217.2075384,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_242.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_242.state')},
'cmd_243': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790307,\n'
' "timestamp_ms" : '
'1666790307183,\n'
' "time" : "Wed Oct 26 15:18:27 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1226106368,\n'
' "io_kbytes" : 1197369,\n'
' "bw_bytes" : 175133033,\n'
' "bw" : 171028,\n'
' "iops" : 342056.277675,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2394736,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 17543,\n'
' "mean" : 182.469767,\n'
' "stddev" : '
'147.807901,\n'
' "N" : 2394736\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5981,\n'
' "max" : 134212,\n'
' "mean" : '
'10884.014790,\n'
' "stddev" : '
'3259.361616,\n'
' "N" : 2394738,\n'
' "percentile" : {\n'
' "1.000000" : 6496,\n'
' "5.000000" : 7392,\n'
' "10.000000" : 7968,\n'
' "20.000000" : 8384,\n'
' "30.000000" : 8768,\n'
' "40.000000" : 9024,\n'
' "50.000000" : 9664,\n'
' "60.000000" : '
'11712,\n'
' "70.000000" : '
'12608,\n'
' "80.000000" : '
'13120,\n'
' "90.000000" : '
'13760,\n'
' "95.000000" : '
'17536,\n'
' "99.000000" : '
'21376,\n'
' "99.500000" : '
'24192,\n'
' "99.900000" : '
'30080,\n'
' "99.950000" : '
'35072,\n'
' "99.990000" : 67072\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6171,\n'
' "max" : 134342,\n'
' "mean" : '
'11066.484526,\n'
' "stddev" : '
'3234.398550,\n'
' "N" : 2394738\n'
' },\n'
' "bw_min" : 167302,\n'
' "bw_max" : 174220,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'171056.571429,\n'
' "bw_dev" : 2202.020531,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 334604,\n'
' "iops_max" : 348440,\n'
' "iops_mean" : '
'342113.142857,\n'
' "iops_stddev" : '
'4404.041061,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 38.157143,\n'
' "sys_cpu" : 61.814286,\n'
' "ctx" : 13,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 5.818804,\n'
' "2" : 42.767136,\n'
' "4" : 51.414060,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 51.271330,\n'
' "20" : 47.557309,\n'
' "50" : 1.134739,\n'
' "100" : 0.035996,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=9088: Wed '
'Oct 26 15:18:27 2022\n'
' read: IOPS=342k, BW=167MiB/s '
'(175MB/s)(1169MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=17543, avg=182.47, '
'stdev=147.81\n'
' clat (usec): min=5, max=134, '
'avg=10.88, stdev= 3.26\n'
' lat (usec): min=6, max=134, '
'avg=11.07, stdev= 3.23\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6496], '
'5.00th=[ 7392], 10.00th=[ 7968], '
'20.00th=[ 8384],\n'
' | 30.00th=[ 8768], '
'40.00th=[ 9024], 50.00th=[ '
'9664], 60.00th=[11712],\n'
' | 70.00th=[12608], '
'80.00th=[13120], '
'90.00th=[13760], '
'95.00th=[17536],\n'
' | 99.00th=[21376], '
'99.50th=[24192], '
'99.90th=[30080], '
'99.95th=[35072],\n'
' | 99.99th=[67072]\n'
' bw ( KiB/s): min=167302, '
'max=174220, per=100.00%, '
'avg=171056.57, stdev=2202.02, '
'samples=14\n'
' iops : min=334604, '
'max=348440, avg=342113.14, '
'stdev=4404.04, samples=14\n'
' lat (usec) : 10=51.27%, '
'20=47.56%, 50=1.13%, 100=0.04%, '
'250=0.01%\n'
' cpu : usr=38.16%, '
'sys=61.81%, ctx=13, majf=0, '
'minf=0\n'
' IO depths : 1=5.8%, '
'2=42.8%, 4=51.4%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2394736,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=167MiB/s (175MB/s), '
'167MiB/s-167MiB/s '
'(175MB/s-175MB/s), io=1169MiB '
'(1226MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_243.output'),
'state': {'begin': 1666783217.2139854,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.8341388702392578,
'end': 1666783218.0481243,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_243.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_243.state')},
'cmd_244': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_244.output'),
'state': {'begin': 1666783220.1547647,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8091392517089844,
'end': 1666783220.963904,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_244.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_244.state')},
'cmd_245': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_245.output'),
'state': {'begin': 1666783220.9659102,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.459925413131714,
'end': 1666783232.4258356,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_245.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_245.state')},
'cmd_246': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790322,\n'
' "timestamp_ms" : '
'1666790322440,\n'
' "time" : "Wed Oct 26 15:18:42 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1559368192,\n'
' "io_kbytes" : 1522820,\n'
' "bw_bytes" : 222735065,\n'
' "bw" : 217514,\n'
' "iops" : 435028.853021,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3045637,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1603,\n'
' "max" : 68709,\n'
' "mean" : 1767.973366,\n'
' "stddev" : '
'121.396579,\n'
' "N" : 3045638\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 220,\n'
' "max" : 119765,\n'
' "mean" : 7169.615502,\n'
' "stddev" : '
'2186.549003,\n'
' "N" : 3045640,\n'
' "percentile" : {\n'
' "1.000000" : 6624,\n'
' "5.000000" : 6688,\n'
' "10.000000" : 6688,\n'
' "20.000000" : 6688,\n'
' "30.000000" : 6752,\n'
' "40.000000" : 6752,\n'
' "50.000000" : 6752,\n'
' "60.000000" : 6816,\n'
' "70.000000" : 6816,\n'
' "80.000000" : 6880,\n'
' "90.000000" : 7072,\n'
' "95.000000" : 7328,\n'
' "99.000000" : '
'18816,\n'
' "99.500000" : '
'21888,\n'
' "99.900000" : '
'25216,\n'
' "99.950000" : '
'32128,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5411,\n'
' "max" : 121538,\n'
' "mean" : 8937.588861,\n'
' "stddev" : '
'2192.642066,\n'
' "N" : 3045640\n'
' },\n'
' "bw_min" : 216461,\n'
' "bw_max" : 219449,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'217555.357143,\n'
' "bw_dev" : 811.925310,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 432922,\n'
' "iops_max" : 438898,\n'
' "iops_mean" : '
'435110.714286,\n'
' "iops_stddev" : '
'1623.850620,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 29.928571,\n'
' "sys_cpu" : 70.042857,\n'
' "ctx" : 17,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 95.836306,\n'
' "20" : 3.568055,\n'
' "50" : 0.557814,\n'
' "100" : 0.036511,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=9145: Wed '
'Oct 26 15:18:42 2022\n'
' read: IOPS=435k, BW=212MiB/s '
'(223MB/s)(1487MiB/7001msec)\n'
' slat (nsec): min=1603, '
'max=68709, avg=1767.97, '
'stdev=121.40\n'
' clat (nsec): min=220, '
'max=119765, avg=7169.62, '
'stdev=2186.55\n'
' lat (usec): min=5, max=121, '
'avg= 8.94, stdev= 2.19\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6624], '
'5.00th=[ 6688], 10.00th=[ 6688], '
'20.00th=[ 6688],\n'
' | 30.00th=[ 6752], '
'40.00th=[ 6752], 50.00th=[ '
'6752], 60.00th=[ 6816],\n'
' | 70.00th=[ 6816], '
'80.00th=[ 6880], 90.00th=[ '
'7072], 95.00th=[ 7328],\n'
' | 99.00th=[18816], '
'99.50th=[21888], '
'99.90th=[25216], '
'99.95th=[32128],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=216461, '
'max=219449, per=100.00%, '
'avg=217555.36, stdev=811.93, '
'samples=14\n'
' iops : min=432922, '
'max=438898, avg=435110.71, '
'stdev=1623.85, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=95.84%, 20=3.57%, 50=0.56%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=29.93%, '
'sys=70.04%, ctx=17, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3045637,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=212MiB/s (223MB/s), '
'212MiB/s-212MiB/s '
'(223MB/s-223MB/s), io=1487MiB '
'(1559MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_246.output'),
'state': {'begin': 1666783232.428395,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.80635666847229,
'end': 1666783233.2347517,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_246.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_246.state')},
'cmd_247': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_247.output'),
'state': {'begin': 1666783235.2788057,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8425042629241943,
'end': 1666783236.12131,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_247.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_247.state')},
'cmd_248': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_248.output'),
'state': {'begin': 1666783236.1232603,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.452204465866089,
'end': 1666783247.5754647,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_248.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_248.state')},
'cmd_249': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790337,\n'
' "timestamp_ms" : '
'1666790337589,\n'
' "time" : "Wed Oct 26 15:18:57 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1562582528,\n'
' "io_kbytes" : 1525959,\n'
' "bw_bytes" : 223194190,\n'
' "bw" : 217963,\n'
' "iops" : 435925.582060,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3051915,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1593,\n'
' "max" : 23705,\n'
' "mean" : 1775.126412,\n'
' "stddev" : '
'116.995816,\n'
' "N" : 3051916\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 231,\n'
' "max" : 117450,\n'
' "mean" : 7161.374734,\n'
' "stddev" : '
'2248.409210,\n'
' "N" : 3051918,\n'
' "percentile" : {\n'
' "1.000000" : 6624,\n'
' "5.000000" : 6624,\n'
' "10.000000" : 6688,\n'
' "20.000000" : 6688,\n'
' "30.000000" : 6752,\n'
' "40.000000" : 6752,\n'
' "50.000000" : 6752,\n'
' "60.000000" : 6752,\n'
' "70.000000" : 6816,\n'
' "80.000000" : 6880,\n'
' "90.000000" : 7072,\n'
' "95.000000" : 7264,\n'
' "99.000000" : '
'19328,\n'
' "99.500000" : '
'24192,\n'
' "99.900000" : '
'25728,\n'
' "99.950000" : '
'32128,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5370,\n'
' "max" : 119194,\n'
' "mean" : 8936.501126,\n'
' "stddev" : '
'2254.690146,\n'
' "N" : 3051918\n'
' },\n'
' "bw_min" : 216970,\n'
' "bw_max" : 219040,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'218076.500000,\n'
' "bw_dev" : 597.526279,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 433940,\n'
' "iops_max" : 438080,\n'
' "iops_mean" : '
'436153.071429,\n'
' "iops_stddev" : '
'1194.975537,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 30.000000,\n'
' "sys_cpu" : 69.985714,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.118994,\n'
' "20" : 3.222043,\n'
' "50" : 0.619414,\n'
' "100" : 0.035486,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=9203: Wed '
'Oct 26 15:18:57 2022\n'
' read: IOPS=436k, BW=213MiB/s '
'(223MB/s)(1490MiB/7001msec)\n'
' slat (nsec): min=1593, '
'max=23705, avg=1775.13, '
'stdev=117.00\n'
' clat (nsec): min=231, '
'max=117450, avg=7161.37, '
'stdev=2248.41\n'
' lat (usec): min=5, max=119, '
'avg= 8.94, stdev= 2.25\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6624], '
'5.00th=[ 6624], 10.00th=[ 6688], '
'20.00th=[ 6688],\n'
' | 30.00th=[ 6752], '
'40.00th=[ 6752], 50.00th=[ '
'6752], 60.00th=[ 6752],\n'
' | 70.00th=[ 6816], '
'80.00th=[ 6880], 90.00th=[ '
'7072], 95.00th=[ 7264],\n'
' | 99.00th=[19328], '
'99.50th=[24192], '
'99.90th=[25728], '
'99.95th=[32128],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=216970, '
'max=219040, per=100.00%, '
'avg=218076.50, stdev=597.53, '
'samples=14\n'
' iops : min=433940, '
'max=438080, avg=436153.07, '
'stdev=1194.98, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.12%, 20=3.22%, 50=0.62%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=30.00%, '
'sys=69.99%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3051915,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=213MiB/s (223MB/s), '
'213MiB/s-213MiB/s '
'(223MB/s-223MB/s), io=1490MiB '
'(1563MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_249.output'),
'state': {'begin': 1666783247.578205,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8199121952056885,
'end': 1666783248.3981173,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_249.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_249.state')},
'cmd_25': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_25.output'),
'state': {'begin': 1666782161.0840642,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8233551979064941,
'end': 1666782161.9074194,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_25.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_25.state')},
'cmd_250': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_250.output'),
'state': {'begin': 1666783250.4902697,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.801767110824585,
'end': 1666783251.2920368,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_250.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_250.state')},
'cmd_251': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_251.output'),
'state': {'begin': 1666783251.293977,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.309634685516357,
'end': 1666783262.6036117,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_251.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_251.state')},
'cmd_252': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790352,\n'
' "timestamp_ms" : '
'1666790352617,\n'
' "time" : "Wed Oct 26 15:19:12 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1559137792,\n'
' "io_kbytes" : 1522595,\n'
' "bw_bytes" : 222702155,\n'
' "bw" : 217482,\n'
' "iops" : 434964.576489,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3045187,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1593,\n'
' "max" : 20638,\n'
' "mean" : 1801.881652,\n'
' "stddev" : '
'112.241295,\n'
' "N" : 3045188\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 211,\n'
' "max" : 117270,\n'
' "mean" : 7169.162602,\n'
' "stddev" : '
'2248.200291,\n'
' "N" : 3045190,\n'
' "percentile" : {\n'
' "1.000000" : 6624,\n'
' "5.000000" : 6624,\n'
' "10.000000" : 6688,\n'
' "20.000000" : 6688,\n'
' "30.000000" : 6752,\n'
' "40.000000" : 6752,\n'
' "50.000000" : 6752,\n'
' "60.000000" : 6816,\n'
' "70.000000" : 6816,\n'
' "80.000000" : 6880,\n'
' "90.000000" : 7072,\n'
' "95.000000" : 7264,\n'
' "99.000000" : '
'19328,\n'
' "99.500000" : '
'24192,\n'
' "99.900000" : '
'25728,\n'
' "99.950000" : '
'32384,\n'
' "99.990000" : 67072\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5360,\n'
' "max" : 118993,\n'
' "mean" : 8971.044281,\n'
' "stddev" : '
'2253.640954,\n'
' "N" : 3045190\n'
' },\n'
' "bw_min" : 216412,\n'
' "bw_max" : 218927,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'217521.785714,\n'
' "bw_dev" : 661.670750,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 432824,\n'
' "iops_max" : 437854,\n'
' "iops_mean" : '
'435043.642857,\n'
' "iops_stddev" : '
'1323.496628,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 32.385714,\n'
' "sys_cpu" : 67.600000,\n'
' "ctx" : 14,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.063559,\n'
' "20" : 3.301308,\n'
' "50" : 0.594709,\n'
' "100" : 0.036911,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=9260: Wed '
'Oct 26 15:19:12 2022\n'
' read: IOPS=435k, BW=212MiB/s '
'(223MB/s)(1487MiB/7001msec)\n'
' slat (nsec): min=1593, '
'max=20638, avg=1801.88, '
'stdev=112.24\n'
' clat (nsec): min=211, '
'max=117270, avg=7169.16, '
'stdev=2248.20\n'
' lat (usec): min=5, max=118, '
'avg= 8.97, stdev= 2.25\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6624], '
'5.00th=[ 6624], 10.00th=[ 6688], '
'20.00th=[ 6688],\n'
' | 30.00th=[ 6752], '
'40.00th=[ 6752], 50.00th=[ '
'6752], 60.00th=[ 6816],\n'
' | 70.00th=[ 6816], '
'80.00th=[ 6880], 90.00th=[ '
'7072], 95.00th=[ 7264],\n'
' | 99.00th=[19328], '
'99.50th=[24192], '
'99.90th=[25728], '
'99.95th=[32384],\n'
' | 99.99th=[67072]\n'
' bw ( KiB/s): min=216412, '
'max=218927, per=100.00%, '
'avg=217521.79, stdev=661.67, '
'samples=14\n'
' iops : min=432824, '
'max=437854, avg=435043.64, '
'stdev=1323.50, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.06%, 20=3.30%, 50=0.59%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=32.39%, '
'sys=67.60%, ctx=14, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3045187,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=212MiB/s (223MB/s), '
'212MiB/s-212MiB/s '
'(223MB/s-223MB/s), io=1487MiB '
'(1559MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_252.output'),
'state': {'begin': 1666783262.6063745,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8146522045135498,
'end': 1666783263.4210267,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_252.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_252.state')},
'cmd_253': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_253.output'),
'state': {'begin': 1666783264.4408336,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7997441291809082,
'end': 1666783265.2405777,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_253.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_253.state')},
'cmd_254': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_254.output'),
'state': {'begin': 1666783265.242522,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.078611850738525,
'end': 1666783276.3211339,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_254.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_254.state')},
'cmd_255': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790366,\n'
' "timestamp_ms" : '
'1666790366377,\n'
' "time" : "Wed Oct 26 15:19:26 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1420984832,\n'
' "io_kbytes" : 1387680,\n'
' "bw_bytes" : 202968837,\n'
' "bw" : 198211,\n'
' "iops" : 396423.082417,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2775358,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1382,\n'
' "max" : 67266,\n'
' "mean" : 1980.791384,\n'
' "stddev" : '
'330.353282,\n'
' "N" : 2775358\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2054,\n'
' "max" : 124162,\n'
' "mean" : 7810.983699,\n'
' "stddev" : '
'2690.900419,\n'
' "N" : 2775361,\n'
' "percentile" : {\n'
' "1.000000" : 4448,\n'
' "5.000000" : 5152,\n'
' "10.000000" : 6816,\n'
' "20.000000" : 6944,\n'
' "30.000000" : 7392,\n'
' "40.000000" : 7456,\n'
' "50.000000" : 7456,\n'
' "60.000000" : 7520,\n'
' "70.000000" : 7648,\n'
' "80.000000" : 8032,\n'
' "90.000000" : 8384,\n'
' "95.000000" : 9408,\n'
' "99.000000" : '
'20864,\n'
' "99.500000" : '
'25216,\n'
' "99.900000" : '
'27264,\n'
' "99.950000" : '
'33536,\n'
' "99.990000" : 69120\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5781,\n'
' "max" : 126286,\n'
' "mean" : 9791.775021,\n'
' "stddev" : '
'2666.740323,\n'
' "N" : 2775361\n'
' },\n'
' "bw_min" : 197094,\n'
' "bw_max" : 199345,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'198313.214286,\n'
' "bw_dev" : 631.854801,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 394188,\n'
' "iops_max" : 398690,\n'
' "iops_mean" : '
'396626.571429,\n'
' "iops_stddev" : '
'1263.775583,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.671429,\n'
' "sys_cpu" : 78.628571,\n'
' "ctx" : 27529,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.050924,\n'
' "20" : 2.666179,\n'
' "50" : 1.239480,\n'
' "100" : 0.041184,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=9307: Wed '
'Oct 26 15:19:26 2022\n'
' read: IOPS=396k, BW=194MiB/s '
'(203MB/s)(1355MiB/7001msec)\n'
' slat (nsec): min=1382, '
'max=67266, avg=1980.79, '
'stdev=330.35\n'
' clat (usec): min=2, max=124, '
'avg= 7.81, stdev= 2.69\n'
' lat (usec): min=5, max=126, '
'avg= 9.79, stdev= 2.67\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4448], '
'5.00th=[ 5152], 10.00th=[ 6816], '
'20.00th=[ 6944],\n'
' | 30.00th=[ 7392], '
'40.00th=[ 7456], 50.00th=[ '
'7456], 60.00th=[ 7520],\n'
' | 70.00th=[ 7648], '
'80.00th=[ 8032], 90.00th=[ '
'8384], 95.00th=[ 9408],\n'
' | 99.00th=[20864], '
'99.50th=[25216], '
'99.90th=[27264], '
'99.95th=[33536],\n'
' | 99.99th=[69120]\n'
' bw ( KiB/s): min=197094, '
'max=199345, per=100.00%, '
'avg=198313.21, stdev=631.85, '
'samples=14\n'
' iops : min=394188, '
'max=398690, avg=396626.57, '
'stdev=1263.78, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=96.05%, 20=2.67%, 50=1.24%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=17.67%, '
'sys=78.63%, ctx=27529, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2775358,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=194MiB/s (203MB/s), '
'194MiB/s-194MiB/s '
'(203MB/s-203MB/s), io=1355MiB '
'(1421MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_255.output'),
'state': {'begin': 1666783276.3240511,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8232474327087402,
'end': 1666783277.1472986,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_255.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_255.state')},
'cmd_256': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_256.output'),
'state': {'begin': 1666783278.1612356,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8103094100952148,
'end': 1666783278.971545,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_256.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_256.state')},
'cmd_257': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_257.output'),
'state': {'begin': 1666783278.9736066,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.077608346939087,
'end': 1666783290.051215,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_257.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_257.state')},
'cmd_258': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790380,\n'
' "timestamp_ms" : '
'1666790380127,\n'
' "time" : "Wed Oct 26 15:19:40 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1402958336,\n'
' "io_kbytes" : 1370076,\n'
' "bw_bytes" : 200393991,\n'
' "bw" : 195697,\n'
' "iops" : 391393.943722,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2740149,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1383,\n'
' "max" : 22943,\n'
' "mean" : 2020.769172,\n'
' "stddev" : '
'287.667604,\n'
' "N" : 2740150\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2103,\n'
' "max" : 122420,\n'
' "mean" : 7908.474523,\n'
' "stddev" : '
'2587.271863,\n'
' "N" : 2740152,\n'
' "percentile" : {\n'
' "1.000000" : 4512,\n'
' "5.000000" : 6816,\n'
' "10.000000" : 6880,\n'
' "20.000000" : 7456,\n'
' "30.000000" : 7456,\n'
' "40.000000" : 7520,\n'
' "50.000000" : 7520,\n'
' "60.000000" : 7584,\n'
' "70.000000" : 7648,\n'
' "80.000000" : 7840,\n'
' "90.000000" : 8256,\n'
' "95.000000" : 9024,\n'
' "99.000000" : '
'20608,\n'
' "99.500000" : '
'24960,\n'
' "99.900000" : '
'27008,\n'
' "99.950000" : '
'33024,\n'
' "99.990000" : 69120\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5590,\n'
' "max" : 123922,\n'
' "mean" : 9929.243570,\n'
' "stddev" : '
'2555.049304,\n'
' "N" : 2740152\n'
' },\n'
' "bw_min" : 194510,\n'
' "bw_max" : 197872,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'195731.571429,\n'
' "bw_dev" : 1047.071059,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 389020,\n'
' "iops_max" : 395744,\n'
' "iops_mean" : '
'391463.214286,\n'
' "iops_stddev" : '
'2094.197044,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 20.785714,\n'
' "sys_cpu" : 75.628571,\n'
' "ctx" : 27189,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.072622,\n'
' "20" : 2.739012,\n'
' "50" : 1.149791,\n'
' "100" : 0.037115,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=9355: Wed '
'Oct 26 15:19:40 2022\n'
' read: IOPS=391k, BW=191MiB/s '
'(200MB/s)(1338MiB/7001msec)\n'
' slat (nsec): min=1383, '
'max=22943, avg=2020.77, '
'stdev=287.67\n'
' clat (usec): min=2, max=122, '
'avg= 7.91, stdev= 2.59\n'
' lat (usec): min=5, max=123, '
'avg= 9.93, stdev= 2.56\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4512], '
'5.00th=[ 6816], 10.00th=[ 6880], '
'20.00th=[ 7456],\n'
' | 30.00th=[ 7456], '
'40.00th=[ 7520], 50.00th=[ '
'7520], 60.00th=[ 7584],\n'
' | 70.00th=[ 7648], '
'80.00th=[ 7840], 90.00th=[ '
'8256], 95.00th=[ 9024],\n'
' | 99.00th=[20608], '
'99.50th=[24960], '
'99.90th=[27008], '
'99.95th=[33024],\n'
' | 99.99th=[69120]\n'
' bw ( KiB/s): min=194510, '
'max=197872, per=100.00%, '
'avg=195731.57, stdev=1047.07, '
'samples=14\n'
' iops : min=389020, '
'max=395744, avg=391463.21, '
'stdev=2094.20, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=96.07%, 20=2.74%, 50=1.15%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=20.79%, '
'sys=75.63%, ctx=27189, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2740149,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=191MiB/s (200MB/s), '
'191MiB/s-191MiB/s '
'(200MB/s-200MB/s), io=1338MiB '
'(1403MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_258.output'),
'state': {'begin': 1666783290.054493,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8424615859985352,
'end': 1666783290.8969545,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_258.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_258.state')},
'cmd_259': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_io_uring-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_259.output'),
'state': {'begin': 1666783291.9385934,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8601806163787842,
'end': 1666783292.798774,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_259.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_259.state')},
'cmd_26': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_26.output'),
'state': {'begin': 1666782161.9096975,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.442041873931885,
'end': 1666782173.3517394,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_26.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_26.state')},
'cmd_260': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_260.output'),
'state': {'begin': 1666783292.8010087,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.117149829864502,
'end': 1666783303.9181585,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_260.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_260.state')},
'cmd_261': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790393,\n'
' "timestamp_ms" : '
'1666790393982,\n'
' "time" : "Wed Oct 26 15:19:53 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1408428544,\n'
' "io_kbytes" : 1375418,\n'
' "bw_bytes" : 201175338,\n'
' "bw" : 196460,\n'
' "iops" : 392920.011427,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2750833,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1402,\n'
' "max" : 43572,\n'
' "mean" : 2012.219970,\n'
' "stddev" : '
'307.976683,\n'
' "N" : 2750834\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 131,\n'
' "max" : 104275,\n'
' "mean" : 7874.613087,\n'
' "stddev" : '
'2621.702740,\n'
' "N" : 2750836,\n'
' "percentile" : {\n'
' "1.000000" : 4448,\n'
' "5.000000" : 6304,\n'
' "10.000000" : 6880,\n'
' "20.000000" : 7200,\n'
' "30.000000" : 7456,\n'
' "40.000000" : 7520,\n'
' "50.000000" : 7520,\n'
' "60.000000" : 7584,\n'
' "70.000000" : 7648,\n'
' "80.000000" : 7904,\n'
' "90.000000" : 8256,\n'
' "95.000000" : 9280,\n'
' "99.000000" : '
'20864,\n'
' "99.500000" : '
'25216,\n'
' "99.900000" : '
'27008,\n'
' "99.950000" : '
'33024,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5771,\n'
' "max" : 106169,\n'
' "mean" : 9886.833346,\n'
' "stddev" : '
'2594.294429,\n'
' "N" : 2750836\n'
' },\n'
' "bw_min" : 195388,\n'
' "bw_max" : 198951,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'196495.071429,\n'
' "bw_dev" : 1111.164217,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 390776,\n'
' "iops_max" : 397903,\n'
' "iops_mean" : '
'392990.214286,\n'
' "iops_stddev" : '
'2222.498462,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 21.085714,\n'
' "sys_cpu" : 75.314286,\n'
' "ctx" : 27181,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.066900,\n'
' "20" : 2.703181,\n'
' "50" : 1.190040,\n'
' "100" : 0.038025,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=9402: Wed '
'Oct 26 15:19:53 2022\n'
' read: IOPS=393k, BW=192MiB/s '
'(201MB/s)(1343MiB/7001msec)\n'
' slat (nsec): min=1402, '
'max=43572, avg=2012.22, '
'stdev=307.98\n'
' clat (nsec): min=131, '
'max=104275, avg=7874.61, '
'stdev=2621.70\n'
' lat (usec): min=5, max=106, '
'avg= 9.89, stdev= 2.59\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4448], '
'5.00th=[ 6304], 10.00th=[ 6880], '
'20.00th=[ 7200],\n'
' | 30.00th=[ 7456], '
'40.00th=[ 7520], 50.00th=[ '
'7520], 60.00th=[ 7584],\n'
' | 70.00th=[ 7648], '
'80.00th=[ 7904], 90.00th=[ '
'8256], 95.00th=[ 9280],\n'
' | 99.00th=[20864], '
'99.50th=[25216], '
'99.90th=[27008], '
'99.95th=[33024],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=195388, '
'max=198951, per=100.00%, '
'avg=196495.07, stdev=1111.16, '
'samples=14\n'
' iops : min=390776, '
'max=397903, avg=392990.21, '
'stdev=2222.50, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.07%, 20=2.70%, 50=1.19%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=21.09%, '
'sys=75.31%, ctx=27181, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2750833,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=192MiB/s (201MB/s), '
'192MiB/s-192MiB/s '
'(201MB/s-201MB/s), io=1343MiB '
'(1408MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_261.output'),
'state': {'begin': 1666783303.9249487,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7905657291412354,
'end': 1666783304.7155144,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_261.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_261.state')},
'cmd_262': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_libaio-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_262.output'),
'state': {'begin': 1666783305.77147,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7845261096954346,
'end': 1666783306.5559962,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_262.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_262.state')},
'cmd_263': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_263.output'),
'state': {'begin': 1666783306.5581977,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.11328387260437,
'end': 1666783317.6714816,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_263.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_263.state')},
'cmd_264': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790407,\n'
' "timestamp_ms" : '
'1666790407746,\n'
' "time" : "Wed Oct 26 15:20:07 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1540268544,\n'
' "io_kbytes" : 1504168,\n'
' "bw_bytes" : 220006933,\n'
' "bw" : 214850,\n'
' "iops" : 429700.471361,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3008333,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1442,\n'
' "max" : 22633,\n'
' "mean" : 1606.975883,\n'
' "stddev" : '
'117.237640,\n'
' "N" : 3008334\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2945,\n'
' "max" : 110618,\n'
' "mean" : 7468.559953,\n'
' "stddev" : '
'2665.426037,\n'
' "N" : 3008336,\n'
' "percentile" : {\n'
' "1.000000" : 6816,\n'
' "5.000000" : 6880,\n'
' "10.000000" : 6880,\n'
' "20.000000" : 6944,\n'
' "30.000000" : 6944,\n'
' "40.000000" : 6944,\n'
' "50.000000" : 7008,\n'
' "60.000000" : 7008,\n'
' "70.000000" : 7072,\n'
' "80.000000" : 7200,\n'
' "90.000000" : 7264,\n'
' "95.000000" : 7392,\n'
' "99.000000" : '
'22400,\n'
' "99.500000" : '
'27008,\n'
' "99.900000" : '
'32640,\n'
' "99.950000" : '
'35072,\n'
' "99.990000" : 69120\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6051,\n'
' "max" : 112181,\n'
' "mean" : 9075.535866,\n'
' "stddev" : '
'2669.379295,\n'
' "N" : 3008336\n'
' },\n'
' "bw_min" : 213880,\n'
' "bw_max" : 215909,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'214911.214286,\n'
' "bw_dev" : 609.982620,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 427760,\n'
' "iops_max" : 431818,\n'
' "iops_mean" : '
'429822.428571,\n'
' "iops_stddev" : '
'1219.965240,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 28.971429,\n'
' "sys_cpu" : 67.200000,\n'
' "ctx" : 26583,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.583922,\n'
' "20" : 2.229574,\n'
' "50" : 1.152033,\n'
' "100" : 0.034438,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 4333182,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 23213,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 23213,\n'
' "util" : 99.069123\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=9449: Wed '
'Oct 26 15:20:07 2022\n'
' read: IOPS=430k, BW=210MiB/s '
'(220MB/s)(1469MiB/7001msec)\n'
' slat (nsec): min=1442, '
'max=22633, avg=1606.98, '
'stdev=117.24\n'
' clat (usec): min=2, max=110, '
'avg= 7.47, stdev= 2.67\n'
' lat (usec): min=6, max=112, '
'avg= 9.08, stdev= 2.67\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6816], '
'5.00th=[ 6880], 10.00th=[ 6880], '
'20.00th=[ 6944],\n'
' | 30.00th=[ 6944], '
'40.00th=[ 6944], 50.00th=[ '
'7008], 60.00th=[ 7008],\n'
' | 70.00th=[ 7072], '
'80.00th=[ 7200], 90.00th=[ '
'7264], 95.00th=[ 7392],\n'
' | 99.00th=[22400], '
'99.50th=[27008], '
'99.90th=[32640], '
'99.95th=[35072],\n'
' | 99.99th=[69120]\n'
' bw ( KiB/s): min=213880, '
'max=215909, per=100.00%, '
'avg=214911.21, stdev=609.98, '
'samples=14\n'
' iops : min=427760, '
'max=431818, avg=429822.43, '
'stdev=1219.97, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=96.58%, 20=2.23%, 50=1.15%, '
'100=0.03%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=28.97%, '
'sys=67.20%, ctx=26583, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3008333,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=210MiB/s (220MB/s), '
'210MiB/s-210MiB/s '
'(220MB/s-220MB/s), io=1469MiB '
'(1540MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=4333182/0, '
'merge=0/0, ticks=23213/0, '
'in_queue=23213, util=99.07%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_264.output'),
'state': {'begin': 1666783317.6742837,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8205270767211914,
'end': 1666783318.4948108,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_264.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_264.state')},
'cmd_265': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_libaio-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_265.output'),
'state': {'begin': 1666783319.5633724,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8108549118041992,
'end': 1666783320.3742273,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_265.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_265.state')},
'cmd_266': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_266.output'),
'state': {'begin': 1666783320.3770204,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.106910943984985,
'end': 1666783331.4839313,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_266.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_266.state')},
'cmd_267': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790421,\n'
' "timestamp_ms" : '
'1666790421560,\n'
' "time" : "Wed Oct 26 15:20:21 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1547104768,\n'
' "io_kbytes" : 1510844,\n'
' "bw_bytes" : 220983397,\n'
' "bw" : 215804,\n'
' "iops" : 431607.627482,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3021685,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1433,\n'
' "max" : 30357,\n'
' "mean" : 1591.780563,\n'
' "stddev" : '
'110.756728,\n'
' "N" : 3021686\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4548,\n'
' "max" : 126658,\n'
' "mean" : 7446.293310,\n'
' "stddev" : '
'2622.424351,\n'
' "N" : 3021688,\n'
' "percentile" : {\n'
' "1.000000" : 6880,\n'
' "5.000000" : 6880,\n'
' "10.000000" : 6944,\n'
' "20.000000" : 6944,\n'
' "30.000000" : 6944,\n'
' "40.000000" : 6944,\n'
' "50.000000" : 7008,\n'
' "60.000000" : 7008,\n'
' "70.000000" : 7008,\n'
' "80.000000" : 7072,\n'
' "90.000000" : 7264,\n'
' "95.000000" : 7392,\n'
' "99.000000" : '
'21120,\n'
' "99.500000" : '
'24960,\n'
' "99.900000" : '
'31872,\n'
' "99.950000" : '
'35072,\n'
' "99.990000" : 70144\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6021,\n'
' "max" : 128201,\n'
' "mean" : 9038.073877,\n'
' "stddev" : '
'2623.736564,\n'
' "N" : 3021688\n'
' },\n'
' "bw_min" : 215056,\n'
' "bw_max" : 217372,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'215893.642857,\n'
' "bw_dev" : 579.696420,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 430112,\n'
' "iops_max" : 434745,\n'
' "iops_mean" : '
'431787.571429,\n'
' "iops_stddev" : '
'1159.639978,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 32.028571,\n'
' "sys_cpu" : 64.071429,\n'
' "ctx" : 29587,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.336084,\n'
' "20" : 2.499665,\n'
' "50" : 1.126226,\n'
' "100" : 0.037760,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 4350489,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 21803,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 21803,\n'
' "util" : 99.078934\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=9497: Wed '
'Oct 26 15:20:21 2022\n'
' read: IOPS=432k, BW=211MiB/s '
'(221MB/s)(1475MiB/7001msec)\n'
' slat (nsec): min=1433, '
'max=30357, avg=1591.78, '
'stdev=110.76\n'
' clat (usec): min=4, max=126, '
'avg= 7.45, stdev= 2.62\n'
' lat (usec): min=6, max=128, '
'avg= 9.04, stdev= 2.62\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6880], '
'5.00th=[ 6880], 10.00th=[ 6944], '
'20.00th=[ 6944],\n'
' | 30.00th=[ 6944], '
'40.00th=[ 6944], 50.00th=[ '
'7008], 60.00th=[ 7008],\n'
' | 70.00th=[ 7008], '
'80.00th=[ 7072], 90.00th=[ '
'7264], 95.00th=[ 7392],\n'
' | 99.00th=[21120], '
'99.50th=[24960], '
'99.90th=[31872], '
'99.95th=[35072],\n'
' | 99.99th=[70144]\n'
' bw ( KiB/s): min=215056, '
'max=217372, per=100.00%, '
'avg=215893.64, stdev=579.70, '
'samples=14\n'
' iops : min=430112, '
'max=434745, avg=431787.57, '
'stdev=1159.64, samples=14\n'
' lat (usec) : 10=96.34%, '
'20=2.50%, 50=1.13%, 100=0.04%, '
'250=0.01%\n'
' cpu : usr=32.03%, '
'sys=64.07%, ctx=29587, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3021685,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=211MiB/s (221MB/s), '
'211MiB/s-211MiB/s '
'(221MB/s-221MB/s), io=1475MiB '
'(1547MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=4350489/0, '
'merge=0/0, ticks=21803/0, '
'in_queue=21803, util=99.08%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_267.output'),
'state': {'begin': 1666783331.4866912,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.869408369064331,
'end': 1666783332.3560996,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_267.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_267.state')},
'cmd_268': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_libaio-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_268.output'),
'state': {'begin': 1666783333.4113686,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.8447458744049072,
'end': 1666783334.2561145,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_268.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_268.state')},
'cmd_269': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_269.output'),
'state': {'begin': 1666783334.2628534,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.113083124160767,
'end': 1666783345.3759365,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_269.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_269.state')},
'cmd_27': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789263,\n'
' "timestamp_ms" : '
'1666789263383,\n'
' "time" : "Wed Oct 26 15:01:03 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 507951104,\n'
' "io_kbytes" : 496046,\n'
' "bw_bytes" : 72564443,\n'
' "bw" : 70863,\n'
' "iops" : 141727.428571,\n'
' "runtime" : 7000,\n'
' "total_ios" : 992092,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 250,\n'
' "max" : 25638,\n'
' "mean" : 278.643752,\n'
' "stddev" : 68.503384,\n'
' "N" : 992092\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 320,\n'
' "max" : 78928,\n'
' "mean" : 6484.535887,\n'
' "stddev" : '
'1090.312041,\n'
' "N" : 992092,\n'
' "percentile" : {\n'
' "1.000000" : 5984,\n'
' "5.000000" : 6176,\n'
' "10.000000" : 6240,\n'
' "20.000000" : 6240,\n'
' "30.000000" : 6304,\n'
' "40.000000" : 6304,\n'
' "50.000000" : 6368,\n'
' "60.000000" : 6368,\n'
' "70.000000" : 6432,\n'
' "80.000000" : 6432,\n'
' "90.000000" : 6496,\n'
' "95.000000" : 6624,\n'
' "99.000000" : 11456,\n'
' "99.500000" : 13504,\n'
' "99.900000" : 17024,\n'
' "99.950000" : 19328,\n'
' "99.990000" : 58112\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5991,\n'
' "max" : 79199,\n'
' "mean" : 6763.179639,\n'
' "stddev" : '
'1092.110784,\n'
' "N" : 992092\n'
' },\n'
' "bw_min" : 70687,\n'
' "bw_max" : 71173,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'70885.071429,\n'
' "bw_dev" : 141.145023,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 141374,\n'
' "iops_max" : 142346,\n'
' "iops_mean" : '
'141770.285714,\n'
' "iops_stddev" : '
'282.276032,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 98.826016,\n'
' "20" : 1.129028,\n'
' "50" : 0.031953,\n'
' "100" : 0.012398,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=5343: Wed Oct 26 15:01:03 '
'2022\n'
' read: IOPS=142k, BW=69.2MiB/s '
'(72.6MB/s)(484MiB/7000msec)\n'
' slat (nsec): min=250, '
'max=25638, avg=278.64, '
'stdev=68.50\n'
' clat (nsec): min=320, '
'max=78928, avg=6484.54, '
'stdev=1090.31\n'
' lat (nsec): min=5991, '
'max=79199, avg=6763.18, '
'stdev=1092.11\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5984], 5.00th=[ '
'6176], 10.00th=[ 6240], 20.00th=[ '
'6240],\n'
' | 30.00th=[ 6304], 40.00th=[ '
'6304], 50.00th=[ 6368], 60.00th=[ '
'6368],\n'
' | 70.00th=[ 6432], 80.00th=[ '
'6432], 90.00th=[ 6496], 95.00th=[ '
'6624],\n'
' | 99.00th=[11456], '
'99.50th=[13504], 99.90th=[17024], '
'99.95th=[19328],\n'
' | 99.99th=[58112]\n'
' bw ( KiB/s): min=70687, '
'max=71173, per=100.00%, '
'avg=70885.07, stdev=141.15, '
'samples=14\n'
' iops : min=141374, '
'max=142346, avg=141770.29, '
'stdev=282.28, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 10=98.83%, '
'20=1.13%, 50=0.03%, 100=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=992092,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=69.2MiB/s (72.6MB/s), '
'69.2MiB/s-69.2MiB/s '
'(72.6MB/s-72.6MB/s), io=484MiB '
'(508MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_27.output'),
'state': {'begin': 1666782173.3538754,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8117146492004395,
'end': 1666782174.16559,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_27.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_27.state')},
'cmd_270': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790435,\n'
' "timestamp_ms" : '
'1666790435451,\n'
' "time" : "Wed Oct 26 15:20:35 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1523316736,\n'
' "io_kbytes" : 1487614,\n'
' "bw_bytes" : 217585592,\n'
' "bw" : 212485,\n'
' "iops" : 424971.289816,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2975224,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1453,\n'
' "max" : 26480,\n'
' "mean" : 1626.616954,\n'
' "stddev" : '
'111.977939,\n'
' "N" : 2975225\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4588,\n'
' "max" : 122941,\n'
' "mean" : 7550.090039,\n'
' "stddev" : '
'2701.313923,\n'
' "N" : 2975227,\n'
' "percentile" : {\n'
' "1.000000" : 6944,\n'
' "5.000000" : 6944,\n'
' "10.000000" : 7008,\n'
' "20.000000" : 7008,\n'
' "30.000000" : 7008,\n'
' "40.000000" : 7072,\n'
' "50.000000" : 7072,\n'
' "60.000000" : 7072,\n'
' "70.000000" : 7136,\n'
' "80.000000" : 7328,\n'
' "90.000000" : 7392,\n'
' "95.000000" : 7456,\n'
' "99.000000" : '
'22400,\n'
' "99.500000" : '
'27008,\n'
' "99.900000" : '
'33024,\n'
' "99.950000" : '
'38656,\n'
' "99.990000" : 71168\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6101,\n'
' "max" : 124514,\n'
' "mean" : 9176.707014,\n'
' "stddev" : '
'2705.260929,\n'
' "N" : 2975227\n'
' },\n'
' "bw_min" : 212046,\n'
' "bw_max" : 213823,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'212606.928571,\n'
' "bw_dev" : 548.418771,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 424092,\n'
' "iops_max" : 427646,\n'
' "iops_mean" : '
'425213.928571,\n'
' "iops_stddev" : '
'1096.881206,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 34.171429,\n'
' "sys_cpu" : 62.028571,\n'
' "ctx" : 26352,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.603953,\n'
' "20" : 2.192776,\n'
' "50" : 1.162669,\n'
' "100" : 0.040568,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 4292890,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 22927,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 22927,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=9543: Wed '
'Oct 26 15:20:35 2022\n'
' read: IOPS=425k, BW=208MiB/s '
'(218MB/s)(1453MiB/7001msec)\n'
' slat (nsec): min=1453, '
'max=26480, avg=1626.62, '
'stdev=111.98\n'
' clat (usec): min=4, max=122, '
'avg= 7.55, stdev= 2.70\n'
' lat (usec): min=6, max=124, '
'avg= 9.18, stdev= 2.71\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6944], '
'5.00th=[ 6944], 10.00th=[ 7008], '
'20.00th=[ 7008],\n'
' | 30.00th=[ 7008], '
'40.00th=[ 7072], 50.00th=[ '
'7072], 60.00th=[ 7072],\n'
' | 70.00th=[ 7136], '
'80.00th=[ 7328], 90.00th=[ '
'7392], 95.00th=[ 7456],\n'
' | 99.00th=[22400], '
'99.50th=[27008], '
'99.90th=[33024], '
'99.95th=[38656],\n'
' | 99.99th=[71168]\n'
' bw ( KiB/s): min=212046, '
'max=213823, per=100.00%, '
'avg=212606.93, stdev=548.42, '
'samples=14\n'
' iops : min=424092, '
'max=427646, avg=425213.93, '
'stdev=1096.88, samples=14\n'
' lat (usec) : 10=96.60%, '
'20=2.19%, 50=1.16%, 100=0.04%, '
'250=0.01%\n'
' cpu : usr=34.17%, '
'sys=62.03%, ctx=26352, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2975224,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=208MiB/s (218MB/s), '
'208MiB/s-208MiB/s '
'(218MB/s-218MB/s), io=1453MiB '
'(1523MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=4292890/0, '
'merge=0/0, ticks=22927/0, '
'in_queue=22927, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_270.output'),
'state': {'begin': 1666783345.3784652,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.8766143321990967,
'end': 1666783346.2550795,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_270.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_270.state')},
'cmd_271': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_libaio-bdev_aio_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_271.output'),
'state': {'begin': 1666783348.498997,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.806602954864502,
'end': 1666783349.3056,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_271.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_271.state')},
'cmd_272': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_272.output'),
'state': {'begin': 1666783349.307994,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-bdev_aio_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.4180166721344,
'end': 1666783360.7260106,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_272.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_272.state')},
'cmd_273': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790450,\n'
' "timestamp_ms" : '
'1666790450686,\n'
' "time" : "Wed Oct 26 15:20:50 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1562015232,\n'
' "io_kbytes" : 1525405,\n'
' "bw_bytes" : 223113159,\n'
' "bw" : 217883,\n'
' "iops" : 435767.318954,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3050807,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1623,\n'
' "max" : 24445,\n'
' "mean" : 1823.933966,\n'
' "stddev" : '
'122.188443,\n'
' "N" : 3050808\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 271,\n'
' "max" : 121558,\n'
' "mean" : 6479.020192,\n'
' "stddev" : '
'2349.697327,\n'
' "N" : 3050807,\n'
' "percentile" : {\n'
' "1.000000" : 4384,\n'
' "5.000000" : 4448,\n'
' "10.000000" : 4576,\n'
' "20.000000" : 4704,\n'
' "30.000000" : 6496,\n'
' "40.000000" : 6624,\n'
' "50.000000" : 6688,\n'
' "60.000000" : 6752,\n'
' "70.000000" : 6752,\n'
' "80.000000" : 6816,\n'
' "90.000000" : 6944,\n'
' "95.000000" : 7136,\n'
' "99.000000" : '
'17536,\n'
' "99.500000" : '
'20352,\n'
' "99.900000" : '
'25472,\n'
' "99.950000" : '
'30848,\n'
' "99.990000" : 65280\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5370,\n'
' "max" : 123371,\n'
' "mean" : 8302.954171,\n'
' "stddev" : '
'2352.126482,\n'
' "N" : 3050807\n'
' },\n'
' "bw_min" : 216213,\n'
' "bw_max" : 219913,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'218014.214286,\n'
' "bw_dev" : 1097.446559,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 432426,\n'
' "iops_max" : 439826,\n'
' "iops_mean" : '
'436028.714286,\n'
' "iops_stddev" : '
'2194.874043,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 38.500000,\n'
' "sys_cpu" : 61.471429,\n'
' "ctx" : 14,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 29.427558,\n'
' "4" : 70.569525,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.818940,\n'
' "20" : 2.569976,\n'
' "50" : 0.567129,\n'
' "100" : 0.034483,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=9600: Wed '
'Oct 26 15:20:50 2022\n'
' read: IOPS=436k, BW=213MiB/s '
'(223MB/s)(1490MiB/7001msec)\n'
' slat (nsec): min=1623, '
'max=24445, avg=1823.93, '
'stdev=122.19\n'
' clat (nsec): min=271, '
'max=121558, avg=6479.02, '
'stdev=2349.70\n'
' lat (usec): min=5, max=123, '
'avg= 8.30, stdev= 2.35\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4384], '
'5.00th=[ 4448], 10.00th=[ 4576], '
'20.00th=[ 4704],\n'
' | 30.00th=[ 6496], '
'40.00th=[ 6624], 50.00th=[ '
'6688], 60.00th=[ 6752],\n'
' | 70.00th=[ 6752], '
'80.00th=[ 6816], 90.00th=[ '
'6944], 95.00th=[ 7136],\n'
' | 99.00th=[17536], '
'99.50th=[20352], '
'99.90th=[25472], '
'99.95th=[30848],\n'
' | 99.99th=[65280]\n'
' bw ( KiB/s): min=216213, '
'max=219913, per=100.00%, '
'avg=218014.21, stdev=1097.45, '
'samples=14\n'
' iops : min=432426, '
'max=439826, avg=436028.71, '
'stdev=2194.87, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.82%, 20=2.57%, 50=0.57%, '
'100=0.03%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=38.50%, '
'sys=61.47%, ctx=14, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=29.4%, 4=70.6%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3050807,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=213MiB/s (223MB/s), '
'213MiB/s-213MiB/s '
'(223MB/s-223MB/s), io=1490MiB '
'(1562MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_273.output'),
'state': {'begin': 1666783360.7284133,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.8341774940490723,
'end': 1666783361.5625908,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_273.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_273.state')},
'cmd_274': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_libaio-bdev_aio_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_274.output'),
'state': {'begin': 1666783363.6808743,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.8134632110595703,
'end': 1666783364.4943376,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_274.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_274.state')},
'cmd_275': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_275.output'),
'state': {'begin': 1666783364.4964976,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-bdev_aio_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.582042932510376,
'end': 1666783376.0785406,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_275.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_275.state')},
'cmd_276': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790466,\n'
' "timestamp_ms" : '
'1666790466054,\n'
' "time" : "Wed Oct 26 15:21:06 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1556518912,\n'
' "io_kbytes" : 1520038,\n'
' "bw_bytes" : 222328083,\n'
' "bw" : 217117,\n'
' "iops" : 434234.109413,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3040073,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1643,\n'
' "max" : 40897,\n'
' "mean" : 1833.893626,\n'
' "stddev" : '
'124.135208,\n'
' "N" : 3040073\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 290,\n'
' "max" : 124533,\n'
' "mean" : 6469.387454,\n'
' "stddev" : '
'2382.210011,\n'
' "N" : 3040076,\n'
' "percentile" : {\n'
' "1.000000" : 4448,\n'
' "5.000000" : 4448,\n'
' "10.000000" : 4576,\n'
' "20.000000" : 4704,\n'
' "30.000000" : 6432,\n'
' "40.000000" : 6624,\n'
' "50.000000" : 6688,\n'
' "60.000000" : 6752,\n'
' "70.000000" : 6816,\n'
' "80.000000" : 6816,\n'
' "90.000000" : 6944,\n'
' "95.000000" : 7200,\n'
' "99.000000" : '
'17536,\n'
' "99.500000" : '
'20352,\n'
' "99.900000" : '
'25728,\n'
' "99.950000" : '
'31872,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5440,\n'
' "max" : 126347,\n'
' "mean" : 8303.281079,\n'
' "stddev" : '
'2384.467555,\n'
' "N" : 3040076\n'
' },\n'
' "bw_min" : 216236,\n'
' "bw_max" : 218096,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'217165.357143,\n'
' "bw_dev" : 551.447969,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 432472,\n'
' "iops_max" : 436190,\n'
' "iops_mean" : '
'434330.785714,\n'
' "iops_stddev" : '
'1102.928562,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 39.585714,\n'
' "sys_cpu" : 60.385714,\n'
' "ctx" : 14,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 31.062050,\n'
' "4" : 68.935220,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 96.805800,\n'
' "20" : 2.581681,\n'
' "50" : 0.566894,\n'
' "100" : 0.036775,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=9659: Wed '
'Oct 26 15:21:06 2022\n'
' read: IOPS=434k, BW=212MiB/s '
'(222MB/s)(1484MiB/7001msec)\n'
' slat (nsec): min=1643, '
'max=40897, avg=1833.89, '
'stdev=124.14\n'
' clat (nsec): min=290, '
'max=124533, avg=6469.39, '
'stdev=2382.21\n'
' lat (usec): min=5, max=126, '
'avg= 8.30, stdev= 2.38\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4448], '
'5.00th=[ 4448], 10.00th=[ 4576], '
'20.00th=[ 4704],\n'
' | 30.00th=[ 6432], '
'40.00th=[ 6624], 50.00th=[ '
'6688], 60.00th=[ 6752],\n'
' | 70.00th=[ 6816], '
'80.00th=[ 6816], 90.00th=[ '
'6944], 95.00th=[ 7200],\n'
' | 99.00th=[17536], '
'99.50th=[20352], '
'99.90th=[25728], '
'99.95th=[31872],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=216236, '
'max=218096, per=100.00%, '
'avg=217165.36, stdev=551.45, '
'samples=14\n'
' iops : min=432472, '
'max=436190, avg=434330.79, '
'stdev=1102.93, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=96.81%, 20=2.58%, '
'50=0.57%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=39.59%, '
'sys=60.39%, ctx=14, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=31.1%, 4=68.9%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3040073,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=212MiB/s (222MB/s), '
'212MiB/s-212MiB/s '
'(222MB/s-222MB/s), io=1484MiB '
'(1557MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_276.output'),
'state': {'begin': 1666783376.0811865,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.830763578414917,
'end': 1666783376.91195,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_276.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_276.state')},
'cmd_277': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_libaio-bdev_aio_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_277.output'),
'state': {'begin': 1666783379.0835202,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.8354787826538086,
'end': 1666783379.918999,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_277.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_277.state')},
'cmd_278': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_278.output'),
'state': {'begin': 1666783379.9209187,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-bdev_aio_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.53004240989685,
'end': 1666783391.450961,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_278.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_278.state')},
'cmd_279': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790481,\n'
' "timestamp_ms" : '
'1666790481430,\n'
' "time" : "Wed Oct 26 15:21:21 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1547431936,\n'
' "io_kbytes" : 1511164,\n'
' "bw_bytes" : 221030129,\n'
' "bw" : 215849,\n'
' "iops" : 431699.042994,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3022325,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1643,\n'
' "max" : 40075,\n'
' "mean" : 1835.594190,\n'
' "stddev" : '
'126.051482,\n'
' "N" : 3022325\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 300,\n'
' "max" : 120215,\n'
' "mean" : 6500.137739,\n'
' "stddev" : '
'2390.829741,\n'
' "N" : 3022328,\n'
' "percentile" : {\n'
' "1.000000" : 4448,\n'
' "5.000000" : 4512,\n'
' "10.000000" : 4576,\n'
' "20.000000" : 4704,\n'
' "30.000000" : 6432,\n'
' "40.000000" : 6688,\n'
' "50.000000" : 6752,\n'
' "60.000000" : 6752,\n'
' "70.000000" : 6816,\n'
' "80.000000" : 6880,\n'
' "90.000000" : 7072,\n'
' "95.000000" : 7264,\n'
' "99.000000" : '
'17536,\n'
' "99.500000" : '
'20608,\n'
' "99.900000" : '
'25984,\n'
' "99.950000" : '
'32128,\n'
' "99.990000" : 64768\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5340,\n'
' "max" : 122139,\n'
' "mean" : 8335.731884,\n'
' "stddev" : '
'2394.300500,\n'
' "N" : 3022328\n'
' },\n'
' "bw_min" : 215090,\n'
' "bw_max" : 216619,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'215940.769231,\n'
' "bw_dev" : 480.123969,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 430180,\n'
' "iops_max" : 433238,\n'
' "iops_mean" : '
'431881.461538,\n'
' "iops_stddev" : '
'960.297056,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 40.442857,\n'
' "sys_cpu" : 59.528571,\n'
' "ctx" : 15,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 31.311093,\n'
' "4" : 68.686789,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.840710,\n'
' "20" : 2.537020,\n'
' "50" : 0.573764,\n'
' "100" : 0.037653,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=9716: Wed '
'Oct 26 15:21:21 2022\n'
' read: IOPS=432k, BW=211MiB/s '
'(221MB/s)(1476MiB/7001msec)\n'
' slat (nsec): min=1643, '
'max=40075, avg=1835.59, '
'stdev=126.05\n'
' clat (nsec): min=300, '
'max=120215, avg=6500.14, '
'stdev=2390.83\n'
' lat (usec): min=5, max=122, '
'avg= 8.34, stdev= 2.39\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4448], '
'5.00th=[ 4512], 10.00th=[ 4576], '
'20.00th=[ 4704],\n'
' | 30.00th=[ 6432], '
'40.00th=[ 6688], 50.00th=[ '
'6752], 60.00th=[ 6752],\n'
' | 70.00th=[ 6816], '
'80.00th=[ 6880], 90.00th=[ '
'7072], 95.00th=[ 7264],\n'
' | 99.00th=[17536], '
'99.50th=[20608], '
'99.90th=[25984], '
'99.95th=[32128],\n'
' | 99.99th=[64768]\n'
' bw ( KiB/s): min=215090, '
'max=216619, per=100.00%, '
'avg=215940.77, stdev=480.12, '
'samples=13\n'
' iops : min=430180, '
'max=433238, avg=431881.46, '
'stdev=960.30, samples=13\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.84%, 20=2.54%, 50=0.57%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=40.44%, '
'sys=59.53%, ctx=15, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=31.3%, 4=68.7%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3022325,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=211MiB/s (221MB/s), '
'211MiB/s-211MiB/s '
'(221MB/s-221MB/s), io=1476MiB '
'(1547MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_279.output'),
'state': {'begin': 1666783391.4535267,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.8779900074005127,
'end': 1666783392.3315167,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_279.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_279.state')},
'cmd_28': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_28.output'),
'state': {'begin': 1666782175.175131,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8835101127624512,
'end': 1666782176.0586412,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_28.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_28.state')},
'cmd_280': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_280.output'),
'state': {'begin': 1666783394.416069,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7993080615997314,
'end': 1666783395.215377,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_280.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_280.state')},
'cmd_281': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_281.output'),
'state': {'begin': 1666783395.2174687,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.530804872512817,
'end': 1666783406.7482736,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_281.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_281.state')},
'cmd_282': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790496,\n'
' "timestamp_ms" : '
'1666790496711,\n'
' "time" : "Wed Oct 26 15:21:36 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1444080640,\n'
' "io_kbytes" : 1410235,\n'
' "bw_bytes" : 206267767,\n'
' "bw" : 201433,\n'
' "iops" : 402866.161977,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2820466,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1673,\n'
' "max" : 34555,\n'
' "mean" : 1859.775083,\n'
' "stddev" : '
'123.879869,\n'
' "N" : 2820467\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 701,\n'
' "max" : 115096,\n'
' "mean" : 6746.611971,\n'
' "stddev" : '
'2343.713145,\n'
' "N" : 2820466,\n'
' "percentile" : {\n'
' "1.000000" : 5216,\n'
' "5.000000" : 5280,\n'
' "10.000000" : 5280,\n'
' "20.000000" : 5344,\n'
' "30.000000" : 5408,\n'
' "40.000000" : 5472,\n'
' "50.000000" : 7264,\n'
' "60.000000" : 7328,\n'
' "70.000000" : 7392,\n'
' "80.000000" : 7456,\n'
' "90.000000" : 7584,\n'
' "95.000000" : 7968,\n'
' "99.000000" : '
'17536,\n'
' "99.500000" : '
'20608,\n'
' "99.900000" : '
'26240,\n'
' "99.950000" : '
'32128,\n'
' "99.990000" : 63744\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5801,\n'
' "max" : 116919,\n'
' "mean" : 8606.387073,\n'
' "stddev" : '
'2346.267108,\n'
' "N" : 2820466\n'
' },\n'
' "bw_min" : 200600,\n'
' "bw_max" : 202229,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'201469.857143,\n'
' "bw_dev" : 500.196093,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 401200,\n'
' "iops_max" : 404458,\n'
' "iops_mean" : '
'402939.714286,\n'
' "iops_stddev" : '
'1000.392187,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 37.757143,\n'
' "sys_cpu" : 62.214286,\n'
' "ctx" : 14,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 49.370813,\n'
' "4" : 50.626563,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.743020,\n'
' "20" : 2.692959,\n'
' "50" : 0.519382,\n'
' "100" : 0.038150,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=9774: Wed '
'Oct 26 15:21:36 2022\n'
' read: IOPS=403k, BW=197MiB/s '
'(206MB/s)(1377MiB/7001msec)\n'
' slat (nsec): min=1673, '
'max=34555, avg=1859.78, '
'stdev=123.88\n'
' clat (nsec): min=701, '
'max=115096, avg=6746.61, '
'stdev=2343.71\n'
' lat (usec): min=5, max=116, '
'avg= 8.61, stdev= 2.35\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5216], '
'5.00th=[ 5280], 10.00th=[ 5280], '
'20.00th=[ 5344],\n'
' | 30.00th=[ 5408], '
'40.00th=[ 5472], 50.00th=[ '
'7264], 60.00th=[ 7328],\n'
' | 70.00th=[ 7392], '
'80.00th=[ 7456], 90.00th=[ '
'7584], 95.00th=[ 7968],\n'
' | 99.00th=[17536], '
'99.50th=[20608], '
'99.90th=[26240], '
'99.95th=[32128],\n'
' | 99.99th=[63744]\n'
' bw ( KiB/s): min=200600, '
'max=202229, per=100.00%, '
'avg=201469.86, stdev=500.20, '
'samples=14\n'
' iops : min=401200, '
'max=404458, avg=402939.71, '
'stdev=1000.39, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.74%, 20=2.69%, 50=0.52%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=37.76%, '
'sys=62.21%, ctx=14, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=49.4%, 4=50.6%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2820466,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=197MiB/s (206MB/s), '
'197MiB/s-197MiB/s '
'(206MB/s-206MB/s), io=1377MiB '
'(1444MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_282.output'),
'state': {'begin': 1666783406.7510893,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8057091236114502,
'end': 1666783407.5567985,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_282.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_282.state')},
'cmd_283': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_283.output'),
'state': {'begin': 1666783409.6616611,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8075447082519531,
'end': 1666783410.4692059,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_283.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_283.state')},
'cmd_284': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_284.output'),
'state': {'begin': 1666783410.4711049,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.679587841033936,
'end': 1666783422.1506927,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_284.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_284.state')},
'cmd_285': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790512,\n'
' "timestamp_ms" : '
'1666790512099,\n'
' "time" : "Wed Oct 26 15:21:52 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1448945664,\n'
' "io_kbytes" : 1414986,\n'
' "bw_bytes" : 206962671,\n'
' "bw" : 202111,\n'
' "iops" : 404223.539494,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2829969,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1653,\n'
' "max" : 22372,\n'
' "mean" : 1839.170135,\n'
' "stddev" : '
'121.423768,\n'
' "N" : 2829970\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 691,\n'
' "max" : 120626,\n'
' "mean" : 6717.754872,\n'
' "stddev" : '
'2347.788874,\n'
' "N" : 2829972,\n'
' "percentile" : {\n'
' "1.000000" : 5152,\n'
' "5.000000" : 5280,\n'
' "10.000000" : 5280,\n'
' "20.000000" : 5280,\n'
' "30.000000" : 5344,\n'
' "40.000000" : 5408,\n'
' "50.000000" : 7264,\n'
' "60.000000" : 7328,\n'
' "70.000000" : 7328,\n'
' "80.000000" : 7392,\n'
' "90.000000" : 7584,\n'
' "95.000000" : 7968,\n'
' "99.000000" : '
'17536,\n'
' "99.500000" : '
'20608,\n'
' "99.900000" : '
'26240,\n'
' "99.950000" : '
'32128,\n'
' "99.990000" : 64256\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5520,\n'
' "max" : 122500,\n'
' "mean" : 8556.924986,\n'
' "stddev" : '
'2350.299146,\n'
' "N" : 2829972\n'
' },\n'
' "bw_min" : 201099,\n'
' "bw_max" : 203576,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'202148.357143,\n'
' "bw_dev" : 804.850259,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 402198,\n'
' "iops_max" : 407152,\n'
' "iops_mean" : '
'404296.714286,\n'
' "iops_stddev" : '
'1609.700518,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 39.814286,\n'
' "sys_cpu" : 60.157143,\n'
' "ctx" : 12,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 49.212165,\n'
' "4" : 50.785291,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.734523,\n'
' "20" : 2.704376,\n'
' "50" : 0.516825,\n'
' "100" : 0.037209,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=9831: Wed '
'Oct 26 15:21:52 2022\n'
' read: IOPS=404k, BW=197MiB/s '
'(207MB/s)(1382MiB/7001msec)\n'
' slat (nsec): min=1653, '
'max=22372, avg=1839.17, '
'stdev=121.42\n'
' clat (nsec): min=691, '
'max=120626, avg=6717.75, '
'stdev=2347.79\n'
' lat (usec): min=5, max=122, '
'avg= 8.56, stdev= 2.35\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5152], '
'5.00th=[ 5280], 10.00th=[ 5280], '
'20.00th=[ 5280],\n'
' | 30.00th=[ 5344], '
'40.00th=[ 5408], 50.00th=[ '
'7264], 60.00th=[ 7328],\n'
' | 70.00th=[ 7328], '
'80.00th=[ 7392], 90.00th=[ '
'7584], 95.00th=[ 7968],\n'
' | 99.00th=[17536], '
'99.50th=[20608], '
'99.90th=[26240], '
'99.95th=[32128],\n'
' | 99.99th=[64256]\n'
' bw ( KiB/s): min=201099, '
'max=203576, per=100.00%, '
'avg=202148.36, stdev=804.85, '
'samples=14\n'
' iops : min=402198, '
'max=407152, avg=404296.71, '
'stdev=1609.70, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.73%, 20=2.70%, 50=0.52%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=39.81%, '
'sys=60.16%, ctx=12, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=49.2%, 4=50.8%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2829969,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=197MiB/s (207MB/s), '
'197MiB/s-197MiB/s '
'(207MB/s-207MB/s), io=1382MiB '
'(1449MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_285.output'),
'state': {'begin': 1666783422.153844,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8044538497924805,
'end': 1666783422.958298,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_285.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_285.state')},
'cmd_286': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_286.output'),
'state': {'begin': 1666783425.066735,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7843372821807861,
'end': 1666783425.8510723,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_286.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_286.state')},
'cmd_287': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_287.output'),
'state': {'begin': 1666783425.8533318,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.602382898330688,
'end': 1666783437.4557147,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_287.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_287.state')},
'cmd_288': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790527,\n'
' "timestamp_ms" : '
'1666790527394,\n'
' "time" : "Wed Oct 26 15:22:07 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1440870912,\n'
' "io_kbytes" : 1407100,\n'
' "bw_bytes" : 205809300,\n'
' "bw" : 200985,\n'
' "iops" : 401970.718469,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2814197,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1673,\n'
' "max" : 45495,\n'
' "mean" : 1855.646163,\n'
' "stddev" : '
'123.276425,\n'
' "N" : 2814198\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 682,\n'
' "max" : 118602,\n'
' "mean" : 6813.198220,\n'
' "stddev" : '
'2388.471879,\n'
' "N" : 2814197,\n'
' "percentile" : {\n'
' "1.000000" : 5216,\n'
' "5.000000" : 5280,\n'
' "10.000000" : 5280,\n'
' "20.000000" : 5344,\n'
' "30.000000" : 5344,\n'
' "40.000000" : 5472,\n'
' "50.000000" : 7264,\n'
' "60.000000" : 7328,\n'
' "70.000000" : 7392,\n'
' "80.000000" : 7456,\n'
' "90.000000" : 7904,\n'
' "95.000000" : 8096,\n'
' "99.000000" : '
'17536,\n'
' "99.500000" : '
'21376,\n'
' "99.900000" : '
'26496,\n'
' "99.950000" : '
'32128,\n'
' "99.990000" : 65280\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5480,\n'
' "max" : 120416,\n'
' "mean" : 8668.844394,\n'
' "stddev" : '
'2391.963514,\n'
' "N" : 2814197\n'
' },\n'
' "bw_min" : 200134,\n'
' "bw_max" : 201911,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'201021.500000,\n'
' "bw_dev" : 640.262897,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 400268,\n'
' "iops_max" : 403823,\n'
' "iops_mean" : '
'402043.071429,\n'
' "iops_stddev" : '
'1280.632684,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 40.957143,\n'
' "sys_cpu" : 59.014286,\n'
' "ctx" : 20,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 47.461923,\n'
' "4" : 52.534986,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 96.760248,\n'
' "20" : 2.542715,\n'
' "50" : 0.653970,\n'
' "100" : 0.034681,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=9890: Wed '
'Oct 26 15:22:07 2022\n'
' read: IOPS=402k, BW=196MiB/s '
'(206MB/s)(1374MiB/7001msec)\n'
' slat (nsec): min=1673, '
'max=45495, avg=1855.65, '
'stdev=123.28\n'
' clat (nsec): min=682, '
'max=118602, avg=6813.20, '
'stdev=2388.47\n'
' lat (usec): min=5, max=120, '
'avg= 8.67, stdev= 2.39\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5216], '
'5.00th=[ 5280], 10.00th=[ 5280], '
'20.00th=[ 5344],\n'
' | 30.00th=[ 5344], '
'40.00th=[ 5472], 50.00th=[ '
'7264], 60.00th=[ 7328],\n'
' | 70.00th=[ 7392], '
'80.00th=[ 7456], 90.00th=[ '
'7904], 95.00th=[ 8096],\n'
' | 99.00th=[17536], '
'99.50th=[21376], '
'99.90th=[26496], '
'99.95th=[32128],\n'
' | 99.99th=[65280]\n'
' bw ( KiB/s): min=200134, '
'max=201911, per=100.00%, '
'avg=201021.50, stdev=640.26, '
'samples=14\n'
' iops : min=400268, '
'max=403823, avg=402043.07, '
'stdev=1280.63, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=96.76%, 20=2.54%, '
'50=0.65%\n'
' lat (usec) : 100=0.03%, '
'250=0.01%\n'
' cpu : usr=40.96%, '
'sys=59.01%, ctx=20, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=47.5%, 4=52.5%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2814197,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=196MiB/s (206MB/s), '
'196MiB/s-196MiB/s '
'(206MB/s-206MB/s), io=1374MiB '
'(1441MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_288.output'),
'state': {'begin': 1666783437.45846,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.870962381362915,
'end': 1666783438.3294225,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_288.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_288.state')},
'cmd_289': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_libaio-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_289.output'),
'state': {'begin': 1666783439.4320314,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8543522357940674,
'end': 1666783440.2863836,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_289.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_289.state')},
'cmd_29': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_29.output'),
'state': {'begin': 1666782176.0607202,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.042643070220947,
'end': 1666782187.1033633,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_29.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_29.state')},
'cmd_290': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_290.output'),
'state': {'begin': 1666783440.2887886,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.134579181671143,
'end': 1666783451.4233677,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_290.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_290.state')},
'cmd_291': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790541,\n'
' "timestamp_ms" : '
'1666790541496,\n'
' "time" : "Wed Oct 26 15:22:21 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1465430528,\n'
' "io_kbytes" : 1431084,\n'
' "bw_bytes" : 209317315,\n'
' "bw" : 204411,\n'
' "iops" : 408822.453935,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2862166,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1502,\n'
' "max" : 21721,\n'
' "mean" : 1667.778486,\n'
' "stddev" : '
'106.823115,\n'
' "N" : 2862166\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 511,\n'
' "max" : 149921,\n'
' "mean" : 7891.621895,\n'
' "stddev" : '
'2696.853545,\n'
' "N" : 2862169,\n'
' "percentile" : {\n'
' "1.000000" : 7264,\n'
' "5.000000" : 7328,\n'
' "10.000000" : 7328,\n'
' "20.000000" : 7328,\n'
' "30.000000" : 7392,\n'
' "40.000000" : 7392,\n'
' "50.000000" : 7392,\n'
' "60.000000" : 7456,\n'
' "70.000000" : 7520,\n'
' "80.000000" : 7648,\n'
' "90.000000" : 7776,\n'
' "95.000000" : 7840,\n'
' "99.000000" : '
'22656,\n'
' "99.500000" : '
'27264,\n'
' "99.900000" : '
'33536,\n'
' "99.950000" : '
'38656,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6402,\n'
' "max" : 151595,\n'
' "mean" : 9559.400362,\n'
' "stddev" : '
'2700.206986,\n'
' "N" : 2862169\n'
' },\n'
' "bw_min" : 203864,\n'
' "bw_max" : 205193,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'204511.357143,\n'
' "bw_dev" : 307.759798,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 407728,\n'
' "iops_max" : 410386,\n'
' "iops_mean" : '
'409022.714286,\n'
' "iops_stddev" : '
'615.561711,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 27.228571,\n'
' "sys_cpu" : 69.314286,\n'
' "ctx" : 25260,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.581889,\n'
' "20" : 2.139429,\n'
' "50" : 1.239236,\n'
' "100" : 0.038922,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=9937: Wed '
'Oct 26 15:22:21 2022\n'
' read: IOPS=409k, BW=200MiB/s '
'(209MB/s)(1398MiB/7001msec)\n'
' slat (nsec): min=1502, '
'max=21721, avg=1667.78, '
'stdev=106.82\n'
' clat (nsec): min=511, '
'max=149921, avg=7891.62, '
'stdev=2696.85\n'
' lat (usec): min=6, max=151, '
'avg= 9.56, stdev= 2.70\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7264], '
'5.00th=[ 7328], 10.00th=[ 7328], '
'20.00th=[ 7328],\n'
' | 30.00th=[ 7392], '
'40.00th=[ 7392], 50.00th=[ '
'7392], 60.00th=[ 7456],\n'
' | 70.00th=[ 7520], '
'80.00th=[ 7648], 90.00th=[ '
'7776], 95.00th=[ 7840],\n'
' | 99.00th=[22656], '
'99.50th=[27264], '
'99.90th=[33536], '
'99.95th=[38656],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=203864, '
'max=205193, per=100.00%, '
'avg=204511.36, stdev=307.76, '
'samples=14\n'
' iops : min=407728, '
'max=410386, avg=409022.71, '
'stdev=615.56, samples=14\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 10=96.58%, '
'20=2.14%, 50=1.24%, 100=0.04%, '
'250=0.01%\n'
' cpu : usr=27.23%, '
'sys=69.31%, ctx=25260, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2862166,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=200MiB/s (209MB/s), '
'200MiB/s-200MiB/s '
'(209MB/s-209MB/s), io=1398MiB '
'(1465MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_291.output'),
'state': {'begin': 1666783451.4265172,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8377313613891602,
'end': 1666783452.2642486,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_291.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_291.state')},
'cmd_292': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_libaio-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_292.output'),
'state': {'begin': 1666783453.292761,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.813385009765625,
'end': 1666783454.106146,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_292.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_292.state')},
'cmd_293': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_293.output'),
'state': {'begin': 1666783454.108579,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.093466997146606,
'end': 1666783465.202046,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_293.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_293.state')},
'cmd_294': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790555,\n'
' "timestamp_ms" : '
'1666790555276,\n'
' "time" : "Wed Oct 26 15:22:35 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1449910272,\n'
' "io_kbytes" : 1415928,\n'
' "bw_bytes" : 207100453,\n'
' "bw" : 202246,\n'
' "iops" : 404492.501071,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2831852,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1533,\n'
' "max" : 21871,\n'
' "mean" : 1679.587689,\n'
' "stddev" : '
'109.338587,\n'
' "N" : 2831853\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4919,\n'
' "max" : 90730,\n'
' "mean" : 7972.974219,\n'
' "stddev" : '
'2662.447268,\n'
' "N" : 2831855,\n'
' "percentile" : {\n'
' "1.000000" : 7392,\n'
' "5.000000" : 7392,\n'
' "10.000000" : 7456,\n'
' "20.000000" : 7456,\n'
' "30.000000" : 7456,\n'
' "40.000000" : 7520,\n'
' "50.000000" : 7520,\n'
' "60.000000" : 7520,\n'
' "70.000000" : 7584,\n'
' "80.000000" : 7648,\n'
' "90.000000" : 7840,\n'
' "95.000000" : 7968,\n'
' "99.000000" : '
'22400,\n'
' "99.500000" : '
'27264,\n'
' "99.900000" : '
'33024,\n'
' "99.950000" : '
'36608,\n'
' "99.990000" : 71168\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6472,\n'
' "max" : 92383,\n'
' "mean" : 9652.561871,\n'
' "stddev" : '
'2665.381488,\n'
' "N" : 2831855\n'
' },\n'
' "bw_min" : 201817,\n'
' "bw_max" : 203505,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'202362.923077,\n'
' "bw_dev" : 461.921072,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 403634,\n'
' "iops_max" : 407010,\n'
' "iops_mean" : '
'404726.153846,\n'
' "iops_stddev" : '
'923.775843,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 32.457143,\n'
' "sys_cpu" : 64.142857,\n'
' "ctx" : 25084,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.526902,\n'
' "20" : 2.143403,\n'
' "50" : 1.292864,\n'
' "100" : 0.036937,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=9985: Wed '
'Oct 26 15:22:35 2022\n'
' read: IOPS=404k, BW=198MiB/s '
'(207MB/s)(1383MiB/7001msec)\n'
' slat (nsec): min=1533, '
'max=21871, avg=1679.59, '
'stdev=109.34\n'
' clat (nsec): min=4919, '
'max=90730, avg=7972.97, '
'stdev=2662.45\n'
' lat (nsec): min=6472, '
'max=92383, avg=9652.56, '
'stdev=2665.38\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7392], '
'5.00th=[ 7392], 10.00th=[ 7456], '
'20.00th=[ 7456],\n'
' | 30.00th=[ 7456], '
'40.00th=[ 7520], 50.00th=[ '
'7520], 60.00th=[ 7520],\n'
' | 70.00th=[ 7584], '
'80.00th=[ 7648], 90.00th=[ '
'7840], 95.00th=[ 7968],\n'
' | 99.00th=[22400], '
'99.50th=[27264], '
'99.90th=[33024], '
'99.95th=[36608],\n'
' | 99.99th=[71168]\n'
' bw ( KiB/s): min=201817, '
'max=203505, per=100.00%, '
'avg=202362.92, stdev=461.92, '
'samples=13\n'
' iops : min=403634, '
'max=407010, avg=404726.15, '
'stdev=923.78, samples=13\n'
' lat (usec) : 10=96.53%, '
'20=2.14%, 50=1.29%, 100=0.04%\n'
' cpu : usr=32.46%, '
'sys=64.14%, ctx=25084, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2831852,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=198MiB/s (207MB/s), '
'198MiB/s-198MiB/s '
'(207MB/s-207MB/s), io=1383MiB '
'(1450MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_294.output'),
'state': {'begin': 1666783465.2048554,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8970766067504883,
'end': 1666783466.101932,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_294.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_294.state')},
'cmd_295': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_4_libaio-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_295.output'),
'state': {'begin': 1666783467.3496056,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 1.1307165622711182,
'end': 1666783468.4803221,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_295.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_295.state')},
'cmd_296': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_296.output'),
'state': {'begin': 1666783468.4822228,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.477175951004028,
'end': 1666783479.9593987,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_296.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_296.state')},
'cmd_297': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790570,\n'
' "timestamp_ms" : '
'1666790570033,\n'
' "time" : "Wed Oct 26 15:22:50 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1468516864,\n'
' "io_kbytes" : 1434098,\n'
' "bw_bytes" : 209758157,\n'
' "bw" : 204841,\n'
' "iops" : 409683.330953,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2868193,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1513,\n'
' "max" : 21661,\n'
' "mean" : 1664.947021,\n'
' "stddev" : '
'111.275646,\n'
' "N" : 2868194\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4869,\n'
' "max" : 125125,\n'
' "mean" : 7874.002652,\n'
' "stddev" : '
'2616.785369,\n'
' "N" : 2868196,\n'
' "percentile" : {\n'
' "1.000000" : 7264,\n'
' "5.000000" : 7328,\n'
' "10.000000" : 7328,\n'
' "20.000000" : 7392,\n'
' "30.000000" : 7392,\n'
' "40.000000" : 7392,\n'
' "50.000000" : 7392,\n'
' "60.000000" : 7456,\n'
' "70.000000" : 7456,\n'
' "80.000000" : 7520,\n'
' "90.000000" : 7712,\n'
' "95.000000" : 7840,\n'
' "99.000000" : '
'21888,\n'
' "99.500000" : '
'25984,\n'
' "99.900000" : '
'33024,\n'
' "99.950000" : '
'36608,\n'
' "99.990000" : 70144\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6412,\n'
' "max" : 126818,\n'
' "mean" : 9538.949647,\n'
' "stddev" : '
'2618.618149,\n'
' "N" : 2868196\n'
' },\n'
' "bw_min" : 203662,\n'
' "bw_max" : 205924,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'204950.642857,\n'
' "bw_dev" : 680.498189,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 407324,\n'
' "iops_max" : 411848,\n'
' "iops_mean" : '
'409901.571429,\n'
' "iops_stddev" : '
'1360.938913,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 32.400000,\n'
' "sys_cpu" : 64.171429,\n'
' "ctx" : 25312,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.547025,\n'
' "20" : 2.215506,\n'
' "50" : 1.198664,\n'
' "100" : 0.038526,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=10034: Wed '
'Oct 26 15:22:50 2022\n'
' read: IOPS=410k, BW=200MiB/s '
'(210MB/s)(1400MiB/7001msec)\n'
' slat (nsec): min=1513, '
'max=21661, avg=1664.95, '
'stdev=111.28\n'
' clat (usec): min=4, max=125, '
'avg= 7.87, stdev= 2.62\n'
' lat (usec): min=6, max=126, '
'avg= 9.54, stdev= 2.62\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7264], '
'5.00th=[ 7328], 10.00th=[ 7328], '
'20.00th=[ 7392],\n'
' | 30.00th=[ 7392], '
'40.00th=[ 7392], 50.00th=[ '
'7392], 60.00th=[ 7456],\n'
' | 70.00th=[ 7456], '
'80.00th=[ 7520], 90.00th=[ '
'7712], 95.00th=[ 7840],\n'
' | 99.00th=[21888], '
'99.50th=[25984], '
'99.90th=[33024], '
'99.95th=[36608],\n'
' | 99.99th=[70144]\n'
' bw ( KiB/s): min=203662, '
'max=205924, per=100.00%, '
'avg=204950.64, stdev=680.50, '
'samples=14\n'
' iops : min=407324, '
'max=411848, avg=409901.57, '
'stdev=1360.94, samples=14\n'
' lat (usec) : 10=96.55%, '
'20=2.22%, 50=1.20%, 100=0.04%, '
'250=0.01%\n'
' cpu : usr=32.40%, '
'sys=64.17%, ctx=25312, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2868193,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=200MiB/s (210MB/s), '
'200MiB/s-200MiB/s '
'(210MB/s-210MB/s), io=1400MiB '
'(1469MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_297.output'),
'state': {'begin': 1666783479.962116,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8609395027160645,
'end': 1666783480.8230555,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_297.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_297.state')},
'cmd_298': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_298.output'),
'state': {'begin': 1666783481.8781722,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8223638534545898,
'end': 1666783482.700536,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_298.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_298.state')},
'cmd_299': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_299.output'),
'state': {'begin': 1666783482.7027597,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.055232286453247,
'end': 1666783493.757992,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_299.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_299.state')},
'cmd_30': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789277,\n'
' "timestamp_ms" : '
'1666789277197,\n'
' "time" : "Wed Oct 26 15:01:17 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 392608768,\n'
' "io_kbytes" : 383407,\n'
' "bw_bytes" : 56078955,\n'
' "bw" : 54764,\n'
' "iops" : 109529.067276,\n'
' "runtime" : 7001,\n'
' "total_ios" : 766813,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1443,\n'
' "max" : 17152,\n'
' "mean" : 1576.314540,\n'
' "stddev" : 85.440023,\n'
' "N" : 766814\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 161,\n'
' "max" : 78006,\n'
' "mean" : 7235.024115,\n'
' "stddev" : 971.994924,\n'
' "N" : 766813,\n'
' "percentile" : {\n'
' "1.000000" : 6752,\n'
' "5.000000" : 6944,\n'
' "10.000000" : 7008,\n'
' "20.000000" : 7008,\n'
' "30.000000" : 7072,\n'
' "40.000000" : 7072,\n'
' "50.000000" : 7136,\n'
' "60.000000" : 7200,\n'
' "70.000000" : 7200,\n'
' "80.000000" : 7264,\n'
' "90.000000" : 7328,\n'
' "95.000000" : 7456,\n'
' "99.000000" : 10304,\n'
' "99.500000" : 13504,\n'
' "99.900000" : 15552,\n'
' "99.950000" : 15808,\n'
' "99.990000" : 58112\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7444,\n'
' "max" : 79509,\n'
' "mean" : 8811.338725,\n'
' "stddev" : 977.342601,\n'
' "N" : 766813\n'
' },\n'
' "bw_min" : 54549,\n'
' "bw_max" : 55065,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'54801.642857,\n'
' "bw_dev" : 147.506249,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 109098,\n'
' "iops_max" : 110130,\n'
' "iops_mean" : '
'109603.500000,\n'
' "iops_stddev" : '
'295.184883,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 9.400000,\n'
' "sys_cpu" : 34.500000,\n'
' "ctx" : 766814,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 98.745718,\n'
' "20" : 1.234330,\n'
' "50" : 0.010000,\n'
' "100" : 0.013954,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 1099741,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 5523,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 5523,\n'
' "util" : 99.108381\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=5391: Wed '
'Oct 26 15:01:17 2022\n'
' read: IOPS=110k, BW=53.5MiB/s '
'(56.1MB/s)(374MiB/7001msec)\n'
' slat (nsec): min=1443, '
'max=17152, avg=1576.31, '
'stdev=85.44\n'
' clat (nsec): min=161, '
'max=78006, avg=7235.02, '
'stdev=971.99\n'
' lat (nsec): min=7444, '
'max=79509, avg=8811.34, '
'stdev=977.34\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6752], 5.00th=[ '
'6944], 10.00th=[ 7008], 20.00th=[ '
'7008],\n'
' | 30.00th=[ 7072], 40.00th=[ '
'7072], 50.00th=[ 7136], 60.00th=[ '
'7200],\n'
' | 70.00th=[ 7200], 80.00th=[ '
'7264], 90.00th=[ 7328], 95.00th=[ '
'7456],\n'
' | 99.00th=[10304], '
'99.50th=[13504], 99.90th=[15552], '
'99.95th=[15808],\n'
' | 99.99th=[58112]\n'
' bw ( KiB/s): min=54549, '
'max=55065, per=100.00%, '
'avg=54801.64, stdev=147.51, '
'samples=14\n'
' iops : min=109098, '
'max=110130, avg=109603.50, '
'stdev=295.18, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=98.75%, 20=1.23%, 50=0.01%, '
'100=0.01%\n'
' cpu : usr=9.40%, '
'sys=34.50%, ctx=766814, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=766813,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=53.5MiB/s (56.1MB/s), '
'53.5MiB/s-53.5MiB/s '
'(56.1MB/s-56.1MB/s), io=374MiB '
'(393MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=1099741/0, '
'merge=0/0, ticks=5523/0, '
'in_queue=5523, util=99.11%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_30.output'),
'state': {'begin': 1666782187.1060665,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.9112894535064697,
'end': 1666782188.017356,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_30.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_30.state')},
'cmd_300': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790583,\n'
' "timestamp_ms" : '
'1666790583831,\n'
' "time" : "Wed Oct 26 15:23:03 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1613466624,\n'
' "io_kbytes" : 1575651,\n'
' "bw_bytes" : 230462308,\n'
' "bw" : 225060,\n'
' "iops" : 450120.554207,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3151294,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1172,\n'
' "max" : 51907,\n'
' "mean" : 1784.391308,\n'
' "stddev" : '
'386.919507,\n'
' "N" : 3151295\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5240,\n'
' "max" : 113272,\n'
' "mean" : '
'15718.233979,\n'
' "stddev" : '
'3820.322070,\n'
' "N" : 3151301,\n'
' "percentile" : {\n'
' "1.000000" : 12224,\n'
' "5.000000" : 12736,\n'
' "10.000000" : '
'13120,\n'
' "20.000000" : '
'14272,\n'
' "30.000000" : '
'14784,\n'
' "40.000000" : '
'14912,\n'
' "50.000000" : '
'15040,\n'
' "60.000000" : '
'15424,\n'
' "70.000000" : '
'15680,\n'
' "80.000000" : '
'16320,\n'
' "90.000000" : '
'17280,\n'
' "95.000000" : '
'18048,\n'
' "99.000000" : '
'36608,\n'
' "99.500000" : '
'39168,\n'
' "99.900000" : '
'48384,\n'
' "99.950000" : '
'64256,\n'
' "99.990000" : 79360\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6453,\n'
' "max" : 114505,\n'
' "mean" : '
'17502.625444,\n'
' "stddev" : '
'3781.638897,\n'
' "N" : 3151301\n'
' },\n'
' "bw_min" : 224723,\n'
' "bw_max" : 225795,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'225187.928571,\n'
' "bw_dev" : 301.062085,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 449446,\n'
' "iops_max" : 451591,\n'
' "iops_mean" : '
'450376.071429,\n'
' "iops_stddev" : '
'602.094097,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 16.871429,\n'
' "sys_cpu" : 79.900000,\n'
' "ctx" : 15021,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.044109,\n'
' "20" : 96.321670,\n'
' "50" : 3.541847,\n'
' "100" : 0.092248,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=10082: Wed Oct 26 15:23:03 '
'2022\n'
' read: IOPS=450k, BW=220MiB/s '
'(230MB/s)(1539MiB/7001msec)\n'
' slat (nsec): min=1172, '
'max=51907, avg=1784.39, '
'stdev=386.92\n'
' clat (usec): min=5, max=113, '
'avg=15.72, stdev= 3.82\n'
' lat (usec): min=6, max=114, '
'avg=17.50, stdev= 3.78\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12224], '
'5.00th=[12736], 10.00th=[13120], '
'20.00th=[14272],\n'
' | 30.00th=[14784], '
'40.00th=[14912], '
'50.00th=[15040], '
'60.00th=[15424],\n'
' | 70.00th=[15680], '
'80.00th=[16320], '
'90.00th=[17280], '
'95.00th=[18048],\n'
' | 99.00th=[36608], '
'99.50th=[39168], '
'99.90th=[48384], '
'99.95th=[64256],\n'
' | 99.99th=[79360]\n'
' bw ( KiB/s): min=224723, '
'max=225795, per=100.00%, '
'avg=225187.93, stdev=301.06, '
'samples=14\n'
' iops : min=449446, '
'max=451591, avg=450376.07, '
'stdev=602.09, samples=14\n'
' lat (usec) : 10=0.04%, '
'20=96.32%, 50=3.54%, 100=0.09%, '
'250=0.01%\n'
' cpu : usr=16.87%, '
'sys=79.90%, ctx=15021, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3151294,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=220MiB/s (230MB/s), '
'220MiB/s-220MiB/s '
'(230MB/s-230MB/s), io=1539MiB '
'(1613MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_300.output'),
'state': {'begin': 1666783493.7646198,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8438622951507568,
'end': 1666783494.6084821,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_300.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_300.state')},
'cmd_301': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_301.output'),
'state': {'begin': 1666783495.661168,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8514082431793213,
'end': 1666783496.5125763,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_301.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_301.state')},
'cmd_302': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_302.output'),
'state': {'begin': 1666783496.515712,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.080080032348633,
'end': 1666783507.595792,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_302.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_302.state')},
'cmd_303': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790597,\n'
' "timestamp_ms" : '
'1666790597635,\n'
' "time" : "Wed Oct 26 15:23:17 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1596023808,\n'
' "io_kbytes" : 1558617,\n'
' "bw_bytes" : 227970833,\n'
' "bw" : 222627,\n'
' "iops" : 445254.392230,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3117226,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1192,\n'
' "max" : 26660,\n'
' "mean" : 1791.133760,\n'
' "stddev" : '
'379.567079,\n'
' "N" : 3117227\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1963,\n'
' "max" : 125385,\n'
' "mean" : '
'15887.595758,\n'
' "stddev" : '
'3772.244360,\n'
' "N" : 3117233,\n'
' "percentile" : {\n'
' "1.000000" : 12224,\n'
' "5.000000" : 12864,\n'
' "10.000000" : '
'13248,\n'
' "20.000000" : '
'14528,\n'
' "30.000000" : '
'14912,\n'
' "40.000000" : '
'15168,\n'
' "50.000000" : '
'15296,\n'
' "60.000000" : '
'15680,\n'
' "70.000000" : '
'15936,\n'
' "80.000000" : '
'16512,\n'
' "90.000000" : '
'17536,\n'
' "95.000000" : '
'18048,\n'
' "99.000000" : '
'36608,\n'
' "99.500000" : '
'38656,\n'
' "99.900000" : '
'47872,\n'
' "99.950000" : '
'63744,\n'
' "99.990000" : 79360\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6492,\n'
' "max" : 126838,\n'
' "mean" : '
'17678.729535,\n'
' "stddev" : '
'3741.390408,\n'
' "N" : 3117233\n'
' },\n'
' "bw_min" : 221913,\n'
' "bw_max" : 223653,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'222667.071429,\n'
' "bw_dev" : 489.091224,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 443826,\n'
' "iops_max" : 447306,\n'
' "iops_mean" : '
'445334.142857,\n'
' "iops_stddev" : '
'978.182448,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.685714,\n'
' "sys_cpu" : 79.200000,\n'
' "ctx" : 14928,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.050077,\n'
' "20" : 96.374341,\n'
' "50" : 3.485727,\n'
' "100" : 0.089374,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=10128: Wed Oct 26 15:23:17 '
'2022\n'
' read: IOPS=445k, BW=217MiB/s '
'(228MB/s)(1522MiB/7001msec)\n'
' slat (nsec): min=1192, '
'max=26660, avg=1791.13, '
'stdev=379.57\n'
' clat (nsec): min=1963, '
'max=125385, avg=15887.60, '
'stdev=3772.24\n'
' lat (usec): min=6, max=126, '
'avg=17.68, stdev= 3.74\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12224], '
'5.00th=[12864], 10.00th=[13248], '
'20.00th=[14528],\n'
' | 30.00th=[14912], '
'40.00th=[15168], '
'50.00th=[15296], '
'60.00th=[15680],\n'
' | 70.00th=[15936], '
'80.00th=[16512], '
'90.00th=[17536], '
'95.00th=[18048],\n'
' | 99.00th=[36608], '
'99.50th=[38656], '
'99.90th=[47872], '
'99.95th=[63744],\n'
' | 99.99th=[79360]\n'
' bw ( KiB/s): min=221913, '
'max=223653, per=100.00%, '
'avg=222667.07, stdev=489.09, '
'samples=14\n'
' iops : min=443826, '
'max=447306, avg=445334.14, '
'stdev=978.18, samples=14\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.05%, 20=96.37%, '
'50=3.49%\n'
' lat (usec) : 100=0.09%, '
'250=0.01%\n'
' cpu : usr=17.69%, '
'sys=79.20%, ctx=14928, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3117226,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=217MiB/s (228MB/s), '
'217MiB/s-217MiB/s '
'(228MB/s-228MB/s), io=1522MiB '
'(1596MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_303.output'),
'state': {'begin': 1666783507.5988343,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8206968307495117,
'end': 1666783508.419531,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_303.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_303.state')},
'cmd_304': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_304.output'),
'state': {'begin': 1666783509.4199626,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.8256092071533203,
'end': 1666783510.2455719,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_304.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_304.state')},
'cmd_305': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_305.output'),
'state': {'begin': 1666783510.2464507,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.052099466323853,
'end': 1666783521.2985501,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_305.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_305.state')},
'cmd_306': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790611,\n'
' "timestamp_ms" : '
'1666790611370,\n'
' "time" : "Wed Oct 26 15:23:31 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1604506112,\n'
' "io_kbytes" : 1566900,\n'
' "bw_bytes" : 229182418,\n'
' "bw" : 223810,\n'
' "iops" : 447620.768462,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3133793,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1162,\n'
' "max" : 23243,\n'
' "mean" : 1790.342289,\n'
' "stddev" : '
'400.121691,\n'
' "N" : 3133794\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1874,\n'
' "max" : 130384,\n'
' "mean" : '
'15806.641568,\n'
' "stddev" : '
'3805.133567,\n'
' "N" : 3133800,\n'
' "percentile" : {\n'
' "1.000000" : 12096,\n'
' "5.000000" : 12736,\n'
' "10.000000" : '
'13248,\n'
' "20.000000" : '
'14400,\n'
' "30.000000" : '
'14784,\n'
' "40.000000" : '
'15040,\n'
' "50.000000" : '
'15168,\n'
' "60.000000" : '
'15552,\n'
' "70.000000" : '
'15808,\n'
' "80.000000" : '
'16512,\n'
' "90.000000" : '
'17280,\n'
' "95.000000" : '
'18048,\n'
' "99.000000" : '
'36608,\n'
' "99.500000" : '
'38656,\n'
' "99.900000" : '
'48384,\n'
' "99.950000" : '
'64256,\n'
' "99.990000" : 79360\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6532,\n'
' "max" : 131657,\n'
' "mean" : '
'17596.983862,\n'
' "stddev" : '
'3768.535898,\n'
' "N" : 3133800\n'
' },\n'
' "bw_min" : 222819,\n'
' "bw_max" : 225676,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'223915.384615,\n'
' "bw_dev" : 891.737867,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 445638,\n'
' "iops_max" : 451352,\n'
' "iops_mean" : '
'447830.846154,\n'
' "iops_stddev" : '
'1783.386285,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.214286,\n'
' "sys_cpu" : 79.585714,\n'
' "ctx" : 14974,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.049142,\n'
' "20" : 96.312360,\n'
' "50" : 3.546022,\n'
' "100" : 0.092412,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=10175: Wed Oct 26 15:23:31 '
'2022\n'
' read: IOPS=448k, BW=219MiB/s '
'(229MB/s)(1530MiB/7001msec)\n'
' slat (nsec): min=1162, '
'max=23243, avg=1790.34, '
'stdev=400.12\n'
' clat (nsec): min=1874, '
'max=130384, avg=15806.64, '
'stdev=3805.13\n'
' lat (usec): min=6, max=131, '
'avg=17.60, stdev= 3.77\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12096], '
'5.00th=[12736], 10.00th=[13248], '
'20.00th=[14400],\n'
' | 30.00th=[14784], '
'40.00th=[15040], '
'50.00th=[15168], '
'60.00th=[15552],\n'
' | 70.00th=[15808], '
'80.00th=[16512], '
'90.00th=[17280], '
'95.00th=[18048],\n'
' | 99.00th=[36608], '
'99.50th=[38656], '
'99.90th=[48384], '
'99.95th=[64256],\n'
' | 99.99th=[79360]\n'
' bw ( KiB/s): min=222819, '
'max=225676, per=100.00%, '
'avg=223915.38, stdev=891.74, '
'samples=13\n'
' iops : min=445638, '
'max=451352, avg=447830.85, '
'stdev=1783.39, samples=13\n'
' lat (usec) : 2=0.01%, '
'10=0.05%, 20=96.31%, 50=3.55%, '
'100=0.09%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=17.21%, '
'sys=79.59%, ctx=14974, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3133793,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=219MiB/s (229MB/s), '
'219MiB/s-219MiB/s '
'(229MB/s-229MB/s), io=1530MiB '
'(1605MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_306.output'),
'state': {'begin': 1666783521.305068,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.9138171672821045,
'end': 1666783522.2188852,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_306.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_306.state')},
'cmd_307': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_307.output'),
'state': {'begin': 1666783523.3202078,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8594181537628174,
'end': 1666783524.179626,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_307.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_307.state')},
'cmd_308': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_308.output'),
'state': {'begin': 1666783524.1820672,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.05566143989563,
'end': 1666783535.2377286,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_308.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_308.state')},
'cmd_309': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790625,\n'
' "timestamp_ms" : '
'1666790625309,\n'
' "time" : "Wed Oct 26 15:23:45 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1514349056,\n'
' "io_kbytes" : 1478856,\n'
' "bw_bytes" : 216335579,\n'
' "bw" : 211265,\n'
' "iops" : 422529.285714,\n'
' "runtime" : 7000,\n'
' "total_ios" : 2957705,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1232,\n'
' "max" : 23514,\n'
' "mean" : 1841.139255,\n'
' "stddev" : '
'389.694801,\n'
' "N" : 2957706\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 3677,\n'
' "max" : 120536,\n'
' "mean" : '
'16780.929818,\n'
' "stddev" : '
'3696.313221,\n'
' "N" : 2957712,\n'
' "percentile" : {\n'
' "1.000000" : 12608,\n'
' "5.000000" : 13248,\n'
' "10.000000" : '
'14400,\n'
' "20.000000" : '
'15296,\n'
' "30.000000" : '
'15808,\n'
' "40.000000" : '
'16192,\n'
' "50.000000" : '
'16512,\n'
' "60.000000" : '
'16768,\n'
' "70.000000" : '
'17024,\n'
' "80.000000" : '
'17536,\n'
' "90.000000" : '
'17792,\n'
' "95.000000" : '
'18560,\n'
' "99.000000" : '
'36608,\n'
' "99.500000" : '
'38656,\n'
' "99.900000" : '
'50432,\n'
' "99.950000" : '
'66048,\n'
' "99.990000" : 81408\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6672,\n'
' "max" : 121879,\n'
' "mean" : '
'18622.069282,\n'
' "stddev" : '
'3671.329577,\n'
' "N" : 2957712\n'
' },\n'
' "bw_min" : 208536,\n'
' "bw_max" : 213329,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'211349.428571,\n'
' "bw_dev" : 1221.796830,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 417072,\n'
' "iops_max" : 426658,\n'
' "iops_mean" : '
'422699.000000,\n'
' "iops_stddev" : '
'2443.594893,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 20.442857,\n'
' "sys_cpu" : 76.714286,\n'
' "ctx" : 14856,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999983,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.040910,\n'
' "20" : 96.235764,\n'
' "50" : 3.622944,\n'
' "100" : 0.100111,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=10221: Wed '
'Oct 26 15:23:45 2022\n'
' read: IOPS=423k, BW=206MiB/s '
'(216MB/s)(1444MiB/7000msec)\n'
' slat (nsec): min=1232, '
'max=23514, avg=1841.14, '
'stdev=389.69\n'
' clat (usec): min=3, max=120, '
'avg=16.78, stdev= 3.70\n'
' lat (usec): min=6, max=121, '
'avg=18.62, stdev= 3.67\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12608], '
'5.00th=[13248], 10.00th=[14400], '
'20.00th=[15296],\n'
' | 30.00th=[15808], '
'40.00th=[16192], '
'50.00th=[16512], '
'60.00th=[16768],\n'
' | 70.00th=[17024], '
'80.00th=[17536], '
'90.00th=[17792], '
'95.00th=[18560],\n'
' | 99.00th=[36608], '
'99.50th=[38656], '
'99.90th=[50432], '
'99.95th=[66048],\n'
' | 99.99th=[81408]\n'
' bw ( KiB/s): min=208536, '
'max=213329, per=100.00%, '
'avg=211349.43, stdev=1221.80, '
'samples=14\n'
' iops : min=417072, '
'max=426658, avg=422699.00, '
'stdev=2443.59, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.04%, 20=96.24%, 50=3.62%, '
'100=0.10%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=20.44%, '
'sys=76.71%, ctx=14856, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2957705,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=206MiB/s (216MB/s), '
'206MiB/s-206MiB/s '
'(216MB/s-216MB/s), io=1444MiB '
'(1514MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_309.output'),
'state': {'begin': 1666783535.2407825,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8338701725006104,
'end': 1666783536.0746527,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_309.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_309.state')},
'cmd_31': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_31.output'),
'state': {'begin': 1666782189.144373,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8392891883850098,
'end': 1666782189.9836621,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_31.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_31.state')},
'cmd_310': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_310.output'),
'state': {'begin': 1666783537.0844007,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8432741165161133,
'end': 1666783537.9276748,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_310.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_310.state')},
'cmd_311': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_311.output'),
'state': {'begin': 1666783537.9298892,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.05512261390686,
'end': 1666783548.9850118,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_311.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_311.state')},
'cmd_312': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790639,\n'
' "timestamp_ms" : '
'1666790639058,\n'
' "time" : "Wed Oct 26 15:23:59 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1536173056,\n'
' "io_kbytes" : 1500169,\n'
' "bw_bytes" : 219421947,\n'
' "bw" : 214279,\n'
' "iops" : 428557.491787,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3000331,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1222,\n'
' "max" : 23965,\n'
' "mean" : 1803.258623,\n'
' "stddev" : '
'390.548360,\n'
' "N" : 3000331\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4839,\n'
' "max" : 128591,\n'
' "mean" : '
'16544.098031,\n'
' "stddev" : '
'3761.336130,\n'
' "N" : 3000338,\n'
' "percentile" : {\n'
' "1.000000" : 12608,\n'
' "5.000000" : 13120,\n'
' "10.000000" : '
'13760,\n'
' "20.000000" : '
'15040,\n'
' "30.000000" : '
'15552,\n'
' "40.000000" : '
'15936,\n'
' "50.000000" : '
'16192,\n'
' "60.000000" : '
'16512,\n'
' "70.000000" : '
'16768,\n'
' "80.000000" : '
'17280,\n'
' "90.000000" : '
'17536,\n'
' "95.000000" : '
'18560,\n'
' "99.000000" : '
'37120,\n'
' "99.500000" : '
'39168,\n'
' "99.900000" : '
'51456,\n'
' "99.950000" : '
'67072,\n'
' "99.990000" : 81408\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6191,\n'
' "max" : 129934,\n'
' "mean" : '
'18347.356140,\n'
' "stddev" : '
'3737.823304,\n'
' "N" : 3000338\n'
' },\n'
' "bw_min" : 211728,\n'
' "bw_max" : 216886,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'214318.285714,\n'
' "bw_dev" : 1642.356301,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 423456,\n'
' "iops_max" : 433772,\n'
' "iops_mean" : '
'428636.857143,\n'
' "iops_stddev" : '
'3284.857914,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 22.057143,\n'
' "sys_cpu" : 75.014286,\n'
' "ctx" : 14798,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999983,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.043895,\n'
' "20" : 96.254080,\n'
' "50" : 3.599036,\n'
' "100" : 0.101522,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=10267: Wed '
'Oct 26 15:23:59 2022\n'
' read: IOPS=429k, BW=209MiB/s '
'(219MB/s)(1465MiB/7001msec)\n'
' slat (nsec): min=1222, '
'max=23965, avg=1803.26, '
'stdev=390.55\n'
' clat (usec): min=4, max=128, '
'avg=16.54, stdev= 3.76\n'
' lat (usec): min=6, max=129, '
'avg=18.35, stdev= 3.74\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12608], '
'5.00th=[13120], 10.00th=[13760], '
'20.00th=[15040],\n'
' | 30.00th=[15552], '
'40.00th=[15936], '
'50.00th=[16192], '
'60.00th=[16512],\n'
' | 70.00th=[16768], '
'80.00th=[17280], '
'90.00th=[17536], '
'95.00th=[18560],\n'
' | 99.00th=[37120], '
'99.50th=[39168], '
'99.90th=[51456], '
'99.95th=[67072],\n'
' | 99.99th=[81408]\n'
' bw ( KiB/s): min=211728, '
'max=216886, per=100.00%, '
'avg=214318.29, stdev=1642.36, '
'samples=14\n'
' iops : min=423456, '
'max=433772, avg=428636.86, '
'stdev=3284.86, samples=14\n'
' lat (usec) : 10=0.04%, '
'20=96.25%, 50=3.60%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=22.06%, '
'sys=75.01%, ctx=14798, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3000331,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=209MiB/s (219MB/s), '
'209MiB/s-209MiB/s '
'(219MB/s-219MB/s), io=1465MiB '
'(1536MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_312.output'),
'state': {'begin': 1666783548.991464,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.795210599899292,
'end': 1666783549.7866745,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_312.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_312.state')},
'cmd_313': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_313.output'),
'state': {'begin': 1666783550.8153117,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.809546709060669,
'end': 1666783551.6248584,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_313.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_313.state')},
'cmd_314': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_314.output'),
'state': {'begin': 1666783551.6267784,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.101061820983887,
'end': 1666783562.7278402,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_314.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_314.state')},
'cmd_315': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790652,\n'
' "timestamp_ms" : '
'1666790652800,\n'
' "time" : "Wed Oct 26 15:24:12 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1528946176,\n'
' "io_kbytes" : 1493111,\n'
' "bw_bytes" : 218389683,\n'
' "bw" : 213271,\n'
' "iops" : 426541.208399,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2986215,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1222,\n'
' "max" : 37791,\n'
' "mean" : 1814.158787,\n'
' "stddev" : '
'388.864751,\n'
' "N" : 2986216\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 121,\n'
' "max" : 127819,\n'
' "mean" : '
'16622.123058,\n'
' "stddev" : '
'3687.178497,\n'
' "N" : 2986222,\n'
' "percentile" : {\n'
' "1.000000" : 12608,\n'
' "5.000000" : 13248,\n'
' "10.000000" : '
'14144,\n'
' "20.000000" : '
'15296,\n'
' "30.000000" : '
'15680,\n'
' "40.000000" : '
'16064,\n'
' "50.000000" : '
'16192,\n'
' "60.000000" : '
'16512,\n'
' "70.000000" : '
'16768,\n'
' "80.000000" : '
'17280,\n'
' "90.000000" : '
'17536,\n'
' "95.000000" : '
'18560,\n'
' "99.000000" : '
'36608,\n'
' "99.500000" : '
'38656,\n'
' "99.900000" : '
'48896,\n'
' "99.950000" : '
'65280,\n'
' "99.990000" : 80384\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6693,\n'
' "max" : 129142,\n'
' "mean" : '
'18436.281854,\n'
' "stddev" : '
'3660.248095,\n'
' "N" : 2986222\n'
' },\n'
' "bw_min" : 212347,\n'
' "bw_max" : 214606,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'213309.142857,\n'
' "bw_dev" : 669.714733,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 424694,\n'
' "iops_max" : 429212,\n'
' "iops_mean" : '
'426618.285714,\n'
' "iops_stddev" : '
'1339.429467,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 21.257143,\n'
' "sys_cpu" : 75.814286,\n'
' "ctx" : 14684,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999983,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.037707,\n'
' "20" : 96.299061,\n'
' "50" : 3.567292,\n'
' "100" : 0.095405,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=10313: Wed '
'Oct 26 15:24:12 2022\n'
' read: IOPS=427k, BW=208MiB/s '
'(218MB/s)(1458MiB/7001msec)\n'
' slat (nsec): min=1222, '
'max=37791, avg=1814.16, '
'stdev=388.86\n'
' clat (nsec): min=121, '
'max=127819, avg=16622.12, '
'stdev=3687.18\n'
' lat (usec): min=6, max=129, '
'avg=18.44, stdev= 3.66\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12608], '
'5.00th=[13248], 10.00th=[14144], '
'20.00th=[15296],\n'
' | 30.00th=[15680], '
'40.00th=[16064], '
'50.00th=[16192], '
'60.00th=[16512],\n'
' | 70.00th=[16768], '
'80.00th=[17280], '
'90.00th=[17536], '
'95.00th=[18560],\n'
' | 99.00th=[36608], '
'99.50th=[38656], '
'99.90th=[48896], '
'99.95th=[65280],\n'
' | 99.99th=[80384]\n'
' bw ( KiB/s): min=212347, '
'max=214606, per=100.00%, '
'avg=213309.14, stdev=669.71, '
'samples=14\n'
' iops : min=424694, '
'max=429212, avg=426618.29, '
'stdev=1339.43, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 10=0.04%, '
'20=96.30%, 50=3.57%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=21.26%, '
'sys=75.81%, ctx=14684, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2986215,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=208MiB/s (218MB/s), '
'208MiB/s-208MiB/s '
'(218MB/s-218MB/s), io=1458MiB '
'(1529MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_315.output'),
'state': {'begin': 1666783562.7305648,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8288524150848389,
'end': 1666783563.5594172,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_315.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_315.state')},
'cmd_316': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_316.output'),
'state': {'begin': 1666783565.6272268,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8053641319274902,
'end': 1666783566.432591,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_316.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_316.state')},
'cmd_317': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_317.output'),
'state': {'begin': 1666783566.4348829,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.552487850189209,
'end': 1666783577.9873707,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_317.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_317.state')},
'cmd_318': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790667,\n'
' "timestamp_ms" : '
'1666790667990,\n'
' "time" : "Wed Oct 26 15:24:27 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2123516928,\n'
' "io_kbytes" : 2073747,\n'
' "bw_bytes" : 303316230,\n'
' "bw" : 296207,\n'
' "iops" : 592413.512355,\n'
' "runtime" : 7001,\n'
' "total_ios" : 4147487,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 190,\n'
' "max" : 26219,\n'
' "mean" : 278.529163,\n'
' "stddev" : 75.839500,\n'
' "N" : 4147488\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 691,\n'
' "max" : 131106,\n'
' "mean" : '
'12536.847786,\n'
' "stddev" : '
'4087.236657,\n'
' "N" : 4147494,\n'
' "percentile" : {\n'
' "1.000000" : 7008,\n'
' "5.000000" : 7904,\n'
' "10.000000" : 8640,\n'
' "20.000000" : 9664,\n'
' "30.000000" : '
'10688,\n'
' "40.000000" : '
'11456,\n'
' "50.000000" : '
'12096,\n'
' "60.000000" : '
'12608,\n'
' "70.000000" : '
'13504,\n'
' "80.000000" : '
'14656,\n'
' "90.000000" : '
'15936,\n'
' "95.000000" : '
'17280,\n'
' "99.000000" : '
'29568,\n'
' "99.500000" : '
'32128,\n'
' "99.900000" : '
'43776,\n'
' "99.950000" : '
'62720,\n'
' "99.990000" : 76288\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5720,\n'
' "max" : 131396,\n'
' "mean" : '
'12815.376954,\n'
' "stddev" : '
'4084.439124,\n'
' "N" : 4147494\n'
' },\n'
' "bw_min" : 295554,\n'
' "bw_max" : 296921,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'296335.857143,\n'
' "bw_dev" : 340.016483,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 591108,\n'
' "iops_max" : 593843,\n'
' "iops_mean" : '
'592671.857143,\n'
' "iops_stddev" : '
'680.086969,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 21,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.100000,\n'
' "4" : 35.838135,\n'
' "8" : 64.137537,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.802206,\n'
' "8" : 0.197794,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 23.339482,\n'
' "20" : 73.205245,\n'
' "50" : 3.369510,\n'
' "100" : 0.084557,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=10371: Wed Oct 26 15:24:27 '
'2022\n'
' read: IOPS=592k, BW=289MiB/s '
'(303MB/s)(2025MiB/7001msec)\n'
' slat (nsec): min=190, '
'max=26219, avg=278.53, '
'stdev=75.84\n'
' clat (nsec): min=691, '
'max=131106, avg=12536.85, '
'stdev=4087.24\n'
' lat (usec): min=5, max=131, '
'avg=12.82, stdev= 4.08\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7008], '
'5.00th=[ 7904], 10.00th=[ 8640], '
'20.00th=[ 9664],\n'
' | 30.00th=[10688], '
'40.00th=[11456], '
'50.00th=[12096], '
'60.00th=[12608],\n'
' | 70.00th=[13504], '
'80.00th=[14656], '
'90.00th=[15936], '
'95.00th=[17280],\n'
' | 99.00th=[29568], '
'99.50th=[32128], '
'99.90th=[43776], '
'99.95th=[62720],\n'
' | 99.99th=[76288]\n'
' bw ( KiB/s): min=295554, '
'max=296921, per=100.00%, '
'avg=296335.86, stdev=340.02, '
'samples=14\n'
' iops : min=591108, '
'max=593843, avg=592671.86, '
'stdev=680.09, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=23.34%, 20=73.21%, '
'50=3.37%\n'
' lat (usec) : 100=0.08%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=21, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=35.8%, 8=64.1%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=99.8%, 8=0.2%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=4147487,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=289MiB/s (303MB/s), '
'289MiB/s-289MiB/s '
'(303MB/s-303MB/s), io=2025MiB '
'(2124MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_318.output'),
'state': {'begin': 1666783577.9900692,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.9011662006378174,
'end': 1666783578.8912354,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_318.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_318.state')},
'cmd_319': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_319.output'),
'state': {'begin': 1666783581.059943,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8159592151641846,
'end': 1666783581.8759022,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_319.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_319.state')},
'cmd_32': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_32.output'),
'state': {'begin': 1666782189.9855325,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.055187463760376,
'end': 1666782201.04072,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_32.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_32.state')},
'cmd_320': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_320.output'),
'state': {'begin': 1666783581.87793,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.513900995254517,
'end': 1666783593.391831,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_320.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_320.state')},
'cmd_321': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790683,\n'
' "timestamp_ms" : '
'1666790683397,\n'
' "time" : "Wed Oct 26 15:24:43 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1576720896,\n'
' "io_kbytes" : 1539766,\n'
' "bw_bytes" : 225213668,\n'
' "bw" : 219935,\n'
' "iops" : 439869.447222,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3079526,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 170,\n'
' "max" : 30026,\n'
' "mean" : 708.241025,\n'
' "stddev" : '
'279.522443,\n'
' "N" : 3079527\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 711,\n'
' "max" : 200556,\n'
' "mean" : '
'16166.595282,\n'
' "stddev" : '
'5857.765495,\n'
' "N" : 3079533,\n'
' "percentile" : {\n'
' "1.000000" : 7456,\n'
' "5.000000" : 8768,\n'
' "10.000000" : 9408,\n'
' "20.000000" : '
'11072,\n'
' "30.000000" : '
'12480,\n'
' "40.000000" : '
'13632,\n'
' "50.000000" : '
'15168,\n'
' "60.000000" : '
'16768,\n'
' "70.000000" : '
'18816,\n'
' "80.000000" : '
'21376,\n'
' "90.000000" : '
'23936,\n'
' "95.000000" : '
'25728,\n'
' "99.000000" : '
'33024,\n'
' "99.500000" : '
'36096,\n'
' "99.900000" : '
'47360,\n'
' "99.950000" : '
'64256,\n'
' "99.990000" : 77312\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5370,\n'
' "max" : 202119,\n'
' "mean" : '
'16874.835481,\n'
' "stddev" : '
'5927.145293,\n'
' "N" : 3079533\n'
' },\n'
' "bw_min" : 198922,\n'
' "bw_max" : 296648,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'220001.714286,\n'
' "bw_dev" : '
'36653.514516,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 397844,\n'
' "iops_max" : 593296,\n'
' "iops_mean" : '
'440003.428571,\n'
' "iops_stddev" : '
'73307.214011,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 21,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 3.569933,\n'
' "4" : 45.009070,\n'
' "8" : 51.419309,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 97.162062,\n'
' "8" : 2.837938,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 13.485322,\n'
' "20" : 61.459426,\n'
' "50" : 24.963355,\n'
' "100" : 0.090858,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=10431: Wed Oct 26 15:24:43 '
'2022\n'
' read: IOPS=440k, BW=215MiB/s '
'(225MB/s)(1504MiB/7001msec)\n'
' slat (nsec): min=170, '
'max=30026, avg=708.24, '
'stdev=279.52\n'
' clat (nsec): min=711, '
'max=200556, avg=16166.60, '
'stdev=5857.77\n'
' lat (usec): min=5, max=202, '
'avg=16.87, stdev= 5.93\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7456], '
'5.00th=[ 8768], 10.00th=[ 9408], '
'20.00th=[11072],\n'
' | 30.00th=[12480], '
'40.00th=[13632], '
'50.00th=[15168], '
'60.00th=[16768],\n'
' | 70.00th=[18816], '
'80.00th=[21376], '
'90.00th=[23936], '
'95.00th=[25728],\n'
' | 99.00th=[33024], '
'99.50th=[36096], '
'99.90th=[47360], '
'99.95th=[64256],\n'
' | 99.99th=[77312]\n'
' bw ( KiB/s): min=198922, '
'max=296648, per=100.00%, '
'avg=220001.71, stdev=36653.51, '
'samples=14\n'
' iops : min=397844, '
'max=593296, avg=440003.43, '
'stdev=73307.21, samples=14\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 2=0.01%, '
'10=13.49%, 20=61.46%, 50=24.96%, '
'100=0.09%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=21, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=3.6%, '
'4=45.0%, 8=51.4%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=97.2%, 8=2.8%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3079526,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=215MiB/s (225MB/s), '
'215MiB/s-215MiB/s '
'(225MB/s-225MB/s), io=1504MiB '
'(1577MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_321.output'),
'state': {'begin': 1666783593.3944402,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8707568645477295,
'end': 1666783594.265197,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_321.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_321.state')},
'cmd_322': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_322.output'),
'state': {'begin': 1666783596.4025614,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8284289836883545,
'end': 1666783597.2309904,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_322.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_322.state')},
'cmd_323': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_323.output'),
'state': {'begin': 1666783597.2333243,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.516216039657593,
'end': 1666783608.7495403,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_323.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_323.state')},
'cmd_324': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790698,\n'
' "timestamp_ms" : '
'1666790698756,\n'
' "time" : "Wed Oct 26 15:24:58 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1556664832,\n'
' "io_kbytes" : 1520180,\n'
' "bw_bytes" : 222348926,\n'
' "bw" : 217137,\n'
' "iops" : 434274.246536,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3040354,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 631,\n'
' "max" : 14688,\n'
' "mean" : 866.610232,\n'
' "stddev" : '
'144.688161,\n'
' "N" : 3040354\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 6462,\n'
' "max" : 132108,\n'
' "mean" : '
'16360.181632,\n'
' "stddev" : '
'5137.436929,\n'
' "N" : 3040360,\n'
' "percentile" : {\n'
' "1.000000" : 8384,\n'
' "5.000000" : 9536,\n'
' "10.000000" : '
'10304,\n'
' "20.000000" : '
'12224,\n'
' "30.000000" : '
'13376,\n'
' "40.000000" : '
'14528,\n'
' "50.000000" : '
'15680,\n'
' "60.000000" : '
'17024,\n'
' "70.000000" : '
'18816,\n'
' "80.000000" : '
'20096,\n'
' "90.000000" : '
'22144,\n'
' "95.000000" : '
'24960,\n'
' "99.000000" : '
'32640,\n'
' "99.500000" : '
'36096,\n'
' "99.900000" : '
'44800,\n'
' "99.950000" : '
'61696,\n'
' "99.990000" : 76288\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7364,\n'
' "max" : 132779,\n'
' "mean" : '
'17226.791887,\n'
' "stddev" : '
'5119.625326,\n'
' "N" : 3040360\n'
' },\n'
' "bw_min" : 216825,\n'
' "bw_max" : 217381,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'217176.142857,\n'
' "bw_dev" : 164.800045,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 433650,\n'
' "iops_max" : 434762,\n'
' "iops_mean" : '
'434352.357143,\n'
' "iops_stddev" : '
'329.733134,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 16,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 1.185553,\n'
' "4" : 45.805258,\n'
' "8" : 53.008959,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 98.136979,\n'
' "8" : 1.863021,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 8.138000,\n'
' "20" : 70.711042,\n'
' "50" : 21.071198,\n'
' "100" : 0.078215,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=10490: Wed Oct 26 15:24:58 '
'2022\n'
' read: IOPS=434k, BW=212MiB/s '
'(222MB/s)(1485MiB/7001msec)\n'
' slat (nsec): min=631, '
'max=14688, avg=866.61, '
'stdev=144.69\n'
' clat (usec): min=6, max=132, '
'avg=16.36, stdev= 5.14\n'
' lat (usec): min=7, max=132, '
'avg=17.23, stdev= 5.12\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8384], '
'5.00th=[ 9536], 10.00th=[10304], '
'20.00th=[12224],\n'
' | 30.00th=[13376], '
'40.00th=[14528], '
'50.00th=[15680], '
'60.00th=[17024],\n'
' | 70.00th=[18816], '
'80.00th=[20096], '
'90.00th=[22144], '
'95.00th=[24960],\n'
' | 99.00th=[32640], '
'99.50th=[36096], '
'99.90th=[44800], '
'99.95th=[61696],\n'
' | 99.99th=[76288]\n'
' bw ( KiB/s): min=216825, '
'max=217381, per=100.00%, '
'avg=217176.14, stdev=164.80, '
'samples=14\n'
' iops : min=433650, '
'max=434762, avg=434352.36, '
'stdev=329.73, samples=14\n'
' lat (usec) : 10=8.14%, '
'20=70.71%, 50=21.07%, 100=0.08%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=16, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=1.2%, '
'4=45.8%, 8=53.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=98.1%, 8=1.9%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3040354,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=212MiB/s (222MB/s), '
'212MiB/s-212MiB/s '
'(222MB/s-222MB/s), io=1485MiB '
'(1557MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_324.output'),
'state': {'begin': 1666783608.752831,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8473482131958008,
'end': 1666783609.6001792,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_324.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_324.state')},
'cmd_325': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_325.output'),
'state': {'begin': 1666783610.610584,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8282756805419922,
'end': 1666783611.4388597,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_325.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_325.state')},
'cmd_326': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_326.output'),
'state': {'begin': 1666783611.4411962,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.043455839157104,
'end': 1666783622.484652,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_326.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_326.state')},
'cmd_327': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790712,\n'
' "timestamp_ms" : '
'1666790712555,\n'
' "time" : "Wed Oct 26 15:25:12 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1552151040,\n'
' "io_kbytes" : 1515772,\n'
' "bw_bytes" : 221704190,\n'
' "bw" : 216507,\n'
' "iops" : 433014.855021,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3031537,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1332,\n'
' "max" : 31810,\n'
' "mean" : 1862.089115,\n'
' "stddev" : '
'370.306038,\n'
' "N" : 3031538\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2144,\n'
' "max" : 123952,\n'
' "mean" : '
'16334.740680,\n'
' "stddev" : '
'3609.143862,\n'
' "N" : 3031544,\n'
' "percentile" : {\n'
' "1.000000" : 12480,\n'
' "5.000000" : 12992,\n'
' "10.000000" : '
'13248,\n'
' "20.000000" : '
'14784,\n'
' "30.000000" : '
'15296,\n'
' "40.000000" : '
'15552,\n'
' "50.000000" : '
'15936,\n'
' "60.000000" : '
'16192,\n'
' "70.000000" : '
'16768,\n'
' "80.000000" : '
'17280,\n'
' "90.000000" : '
'17792,\n'
' "95.000000" : '
'18816,\n'
' "99.000000" : '
'32640,\n'
' "99.500000" : '
'35072,\n'
' "99.900000" : '
'51456,\n'
' "99.950000" : '
'67072,\n'
' "99.990000" : 78336\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7133,\n'
' "max" : 125345,\n'
' "mean" : '
'18196.829907,\n'
' "stddev" : '
'3573.540743,\n'
' "N" : 3031544\n'
' },\n'
' "bw_min" : 215528,\n'
' "bw_max" : 217965,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'216654.571429,\n'
' "bw_dev" : 805.018749,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 431056,\n'
' "iops_max" : 435931,\n'
' "iops_mean" : '
'433309.285714,\n'
' "iops_stddev" : '
'1610.180087,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 16.757143,\n'
' "sys_cpu" : 80.328571,\n'
' "ctx" : 15022,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999967,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.056506,\n'
' "20" : 95.943741,\n'
' "50" : 3.889281,\n'
' "100" : 0.109647,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 4373344,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 25161,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 25161,\n'
' "util" : 99.108381\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=10537: Wed '
'Oct 26 15:25:12 2022\n'
' read: IOPS=433k, BW=211MiB/s '
'(222MB/s)(1480MiB/7001msec)\n'
' slat (nsec): min=1332, '
'max=31810, avg=1862.09, '
'stdev=370.31\n'
' clat (usec): min=2, max=123, '
'avg=16.33, stdev= 3.61\n'
' lat (usec): min=7, max=125, '
'avg=18.20, stdev= 3.57\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12480], '
'5.00th=[12992], 10.00th=[13248], '
'20.00th=[14784],\n'
' | 30.00th=[15296], '
'40.00th=[15552], '
'50.00th=[15936], '
'60.00th=[16192],\n'
' | 70.00th=[16768], '
'80.00th=[17280], '
'90.00th=[17792], '
'95.00th=[18816],\n'
' | 99.00th=[32640], '
'99.50th=[35072], '
'99.90th=[51456], '
'99.95th=[67072],\n'
' | 99.99th=[78336]\n'
' bw ( KiB/s): min=215528, '
'max=217965, per=100.00%, '
'avg=216654.57, stdev=805.02, '
'samples=14\n'
' iops : min=431056, '
'max=435931, avg=433309.29, '
'stdev=1610.18, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.06%, 20=95.94%, 50=3.89%, '
'100=0.11%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=16.76%, '
'sys=80.33%, ctx=15022, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3031537,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=211MiB/s (222MB/s), '
'211MiB/s-211MiB/s '
'(222MB/s-222MB/s), io=1480MiB '
'(1552MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=4373344/0, '
'merge=0/0, ticks=25161/0, '
'in_queue=25161, util=99.11%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_327.output'),
'state': {'begin': 1666783622.4870234,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.860776424407959,
'end': 1666783623.3477998,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_327.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_327.state')},
'cmd_328': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_328.output'),
'state': {'begin': 1666783624.3991337,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.850975513458252,
'end': 1666783625.2501092,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_328.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_328.state')},
'cmd_329': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_329.output'),
'state': {'begin': 1666783625.2517607,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.097941875457764,
'end': 1666783636.3497026,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_329.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_329.state')},
'cmd_33': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789291,\n'
' "timestamp_ms" : '
'1666789291136,\n'
' "time" : "Wed Oct 26 15:01:31 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 392546816,\n'
' "io_kbytes" : 383346,\n'
' "bw_bytes" : 56070106,\n'
' "bw" : 54755,\n'
' "iops" : 109511.784031,\n'
' "runtime" : 7001,\n'
' "total_ios" : 766692,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1452,\n'
' "max" : 7204,\n'
' "mean" : 1551.774812,\n'
' "stddev" : 73.369744,\n'
' "N" : 766693\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1042,\n'
' "max" : 78077,\n'
' "mean" : 7262.353467,\n'
' "stddev" : 979.853703,\n'
' "N" : 766692,\n'
' "percentile" : {\n'
' "1.000000" : 6816,\n'
' "5.000000" : 6944,\n'
' "10.000000" : 7008,\n'
' "20.000000" : 7072,\n'
' "30.000000" : 7072,\n'
' "40.000000" : 7136,\n'
' "50.000000" : 7136,\n'
' "60.000000" : 7200,\n'
' "70.000000" : 7264,\n'
' "80.000000" : 7264,\n'
' "90.000000" : 7328,\n'
' "95.000000" : 7456,\n'
' "99.000000" : 10432,\n'
' "99.500000" : 13632,\n'
' "99.900000" : 15680,\n'
' "99.950000" : 15808,\n'
' "99.990000" : 58112\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6743,\n'
' "max" : 79590,\n'
' "mean" : 8814.128316,\n'
' "stddev" : 984.427864,\n'
' "N" : 766692\n'
' },\n'
' "bw_min" : 54595,\n'
' "bw_max" : 55137,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'54770.357143,\n'
' "bw_dev" : 141.769915,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 109190,\n'
' "iops_max" : 110274,\n'
' "iops_mean" : '
'109540.714286,\n'
' "iops_stddev" : '
'283.539830,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 8.571429,\n'
' "sys_cpu" : 34.971429,\n'
' "ctx" : 766689,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 98.723738,\n'
' "20" : 1.253959,\n'
' "50" : 0.010000,\n'
' "100" : 0.014217,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 1098692,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 5507,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 5507,\n'
' "util" : 99.069123\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=5437: Wed '
'Oct 26 15:01:31 2022\n'
' read: IOPS=110k, BW=53.5MiB/s '
'(56.1MB/s)(374MiB/7001msec)\n'
' slat (nsec): min=1452, '
'max=7204, avg=1551.77, '
'stdev=73.37\n'
' clat (nsec): min=1042, '
'max=78077, avg=7262.35, '
'stdev=979.85\n'
' lat (nsec): min=6743, '
'max=79590, avg=8814.13, '
'stdev=984.43\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6816], 5.00th=[ '
'6944], 10.00th=[ 7008], 20.00th=[ '
'7072],\n'
' | 30.00th=[ 7072], 40.00th=[ '
'7136], 50.00th=[ 7136], 60.00th=[ '
'7200],\n'
' | 70.00th=[ 7264], 80.00th=[ '
'7264], 90.00th=[ 7328], 95.00th=[ '
'7456],\n'
' | 99.00th=[10432], '
'99.50th=[13632], 99.90th=[15680], '
'99.95th=[15808],\n'
' | 99.99th=[58112]\n'
' bw ( KiB/s): min=54595, '
'max=55137, per=100.00%, '
'avg=54770.36, stdev=141.77, '
'samples=14\n'
' iops : min=109190, '
'max=110274, avg=109540.71, '
'stdev=283.54, samples=14\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=98.72%, 20=1.25%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%\n'
' cpu : usr=8.57%, '
'sys=34.97%, ctx=766689, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=766692,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=53.5MiB/s (56.1MB/s), '
'53.5MiB/s-53.5MiB/s '
'(56.1MB/s-56.1MB/s), io=374MiB '
'(393MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=1098692/0, '
'merge=0/0, ticks=5507/0, '
'in_queue=5507, util=99.07%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_33.output'),
'state': {'begin': 1666782201.0473912,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8552923202514648,
'end': 1666782201.9026835,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_33.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_33.state')},
'cmd_330': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790726,\n'
' "timestamp_ms" : '
'1666790726420,\n'
' "time" : "Wed Oct 26 15:25:26 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1511606784,\n'
' "io_kbytes" : 1476178,\n'
' "bw_bytes" : 215912981,\n'
' "bw" : 210852,\n'
' "iops" : 421704.042280,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2952350,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1333,\n'
' "max" : 30808,\n'
' "mean" : 1878.538164,\n'
' "stddev" : '
'370.577820,\n'
' "N" : 2952350\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 3927,\n'
' "max" : 150492,\n'
' "mean" : '
'16780.232220,\n'
' "stddev" : '
'3545.257165,\n'
' "N" : 2952357,\n'
' "percentile" : {\n'
' "1.000000" : 12736,\n'
' "5.000000" : 13248,\n'
' "10.000000" : '
'13760,\n'
' "20.000000" : '
'15168,\n'
' "30.000000" : '
'15808,\n'
' "40.000000" : '
'16192,\n'
' "50.000000" : '
'16512,\n'
' "60.000000" : '
'16768,\n'
' "70.000000" : '
'17024,\n'
' "80.000000" : '
'17536,\n'
' "90.000000" : '
'18048,\n'
' "95.000000" : '
'18816,\n'
' "99.000000" : '
'33024,\n'
' "99.500000" : '
'35072,\n'
' "99.900000" : '
'50944,\n'
' "99.950000" : '
'67072,\n'
' "99.990000" : 78336\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7194,\n'
' "max" : 151965,\n'
' "mean" : '
'18658.769599,\n'
' "stddev" : '
'3523.940859,\n'
' "N" : 2952357\n'
' },\n'
' "bw_min" : 208714,\n'
' "bw_max" : 212848,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'210911.285714,\n'
' "bw_dev" : 1029.007022,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 417428,\n'
' "iops_max" : 425697,\n'
' "iops_mean" : '
'421822.500000,\n'
' "iops_stddev" : '
'2058.121277,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.842857,\n'
' "sys_cpu" : 81.442857,\n'
' "ctx" : 15090,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999966,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.087591,\n'
' "20" : 95.854726,\n'
' "50" : 3.952004,\n'
' "100" : 0.104730,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 4279306,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 24601,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 24601,\n'
' "util" : 99.069123\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=10583: Wed '
'Oct 26 15:25:26 2022\n'
' read: IOPS=422k, BW=206MiB/s '
'(216MB/s)(1442MiB/7001msec)\n'
' slat (nsec): min=1333, '
'max=30808, avg=1878.54, '
'stdev=370.58\n'
' clat (usec): min=3, max=150, '
'avg=16.78, stdev= 3.55\n'
' lat (usec): min=7, max=151, '
'avg=18.66, stdev= 3.52\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12736], '
'5.00th=[13248], 10.00th=[13760], '
'20.00th=[15168],\n'
' | 30.00th=[15808], '
'40.00th=[16192], '
'50.00th=[16512], '
'60.00th=[16768],\n'
' | 70.00th=[17024], '
'80.00th=[17536], '
'90.00th=[18048], '
'95.00th=[18816],\n'
' | 99.00th=[33024], '
'99.50th=[35072], '
'99.90th=[50944], '
'99.95th=[67072],\n'
' | 99.99th=[78336]\n'
' bw ( KiB/s): min=208714, '
'max=212848, per=100.00%, '
'avg=210911.29, stdev=1029.01, '
'samples=14\n'
' iops : min=417428, '
'max=425697, avg=421822.50, '
'stdev=2058.12, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.09%, 20=95.85%, 50=3.95%, '
'100=0.10%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=15.84%, '
'sys=81.44%, ctx=15090, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2952350,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=206MiB/s (216MB/s), '
'206MiB/s-206MiB/s '
'(216MB/s-216MB/s), io=1442MiB '
'(1512MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=4279306/0, '
'merge=0/0, ticks=24601/0, '
'in_queue=24601, util=99.07%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_330.output'),
'state': {'begin': 1666783636.3520877,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8014988899230957,
'end': 1666783637.1535866,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_330.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_330.state')},
'cmd_331': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_331.output'),
'state': {'begin': 1666783638.214614,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.8205971717834473,
'end': 1666783639.035211,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_331.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_331.state')},
'cmd_332': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_332.output'),
'state': {'begin': 1666783639.0374577,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.052669763565063,
'end': 1666783650.0901275,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_332.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_332.state')},
'cmd_333': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790740,\n'
' "timestamp_ms" : '
'1666790740160,\n'
' "time" : "Wed Oct 26 15:25:40 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1562410496,\n'
' "io_kbytes" : 1525791,\n'
' "bw_bytes" : 223169618,\n'
' "bw" : 217939,\n'
' "iops" : 435877.017569,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3051575,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1322,\n'
' "max" : 37350,\n'
' "mean" : 1848.436398,\n'
' "stddev" : '
'360.896377,\n'
' "N" : 3051576\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5079,\n'
' "max" : 131407,\n'
' "mean" : '
'16231.104474,\n'
' "stddev" : '
'3574.667520,\n'
' "N" : 3051582,\n'
' "percentile" : {\n'
' "1.000000" : 12480,\n'
' "5.000000" : 12992,\n'
' "10.000000" : '
'13248,\n'
' "20.000000" : '
'14784,\n'
' "30.000000" : '
'15296,\n'
' "40.000000" : '
'15424,\n'
' "50.000000" : '
'15808,\n'
' "60.000000" : '
'16064,\n'
' "70.000000" : '
'16512,\n'
' "80.000000" : '
'17280,\n'
' "90.000000" : '
'17792,\n'
' "95.000000" : '
'18560,\n'
' "99.000000" : '
'32640,\n'
' "99.500000" : '
'34560,\n'
' "99.900000" : '
'49408,\n'
' "99.950000" : '
'65280,\n'
' "99.990000" : 77312\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6452,\n'
' "max" : 132769,\n'
' "mean" : '
'18079.540984,\n'
' "stddev" : '
'3537.875972,\n'
' "N" : 3051582\n'
' },\n'
' "bw_min" : 217144,\n'
' "bw_max" : 219141,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'217999.642857,\n'
' "bw_dev" : 593.977934,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 434288,\n'
' "iops_max" : 438282,\n'
' "iops_mean" : '
'435999.357143,\n'
' "iops_stddev" : '
'1188.013701,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 16.628571,\n'
' "sys_cpu" : 80.442857,\n'
' "ctx" : 14900,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999967,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.058167,\n'
' "20" : 96.003277,\n'
' "50" : 3.840115,\n'
' "100" : 0.098146,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 4399333,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 25278,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 25278,\n'
' "util" : 99.069123\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=10630: Wed '
'Oct 26 15:25:40 2022\n'
' read: IOPS=436k, BW=213MiB/s '
'(223MB/s)(1490MiB/7001msec)\n'
' slat (nsec): min=1322, '
'max=37350, avg=1848.44, '
'stdev=360.90\n'
' clat (usec): min=5, max=131, '
'avg=16.23, stdev= 3.57\n'
' lat (usec): min=6, max=132, '
'avg=18.08, stdev= 3.54\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12480], '
'5.00th=[12992], 10.00th=[13248], '
'20.00th=[14784],\n'
' | 30.00th=[15296], '
'40.00th=[15424], '
'50.00th=[15808], '
'60.00th=[16064],\n'
' | 70.00th=[16512], '
'80.00th=[17280], '
'90.00th=[17792], '
'95.00th=[18560],\n'
' | 99.00th=[32640], '
'99.50th=[34560], '
'99.90th=[49408], '
'99.95th=[65280],\n'
' | 99.99th=[77312]\n'
' bw ( KiB/s): min=217144, '
'max=219141, per=100.00%, '
'avg=217999.64, stdev=593.98, '
'samples=14\n'
' iops : min=434288, '
'max=438282, avg=435999.36, '
'stdev=1188.01, samples=14\n'
' lat (usec) : 10=0.06%, '
'20=96.00%, 50=3.84%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=16.63%, '
'sys=80.44%, ctx=14900, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3051575,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=213MiB/s (223MB/s), '
'213MiB/s-213MiB/s '
'(223MB/s-223MB/s), io=1490MiB '
'(1562MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=4399333/0, '
'merge=0/0, ticks=25278/0, '
'in_queue=25278, util=99.07%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_333.output'),
'state': {'begin': 1666783650.092708,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.842928409576416,
'end': 1666783650.9356365,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_333.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_333.state')},
'cmd_334': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_334.output'),
'state': {'begin': 1666783653.055604,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.8298826217651367,
'end': 1666783653.8854866,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_334.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_334.state')},
'cmd_335': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_335.output'),
'state': {'begin': 1666783653.8873825,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.41184949874878,
'end': 1666783665.299232,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_335.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_335.state')},
'cmd_336': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790755,\n'
' "timestamp_ms" : '
'1666790755304,\n'
' "time" : "Wed Oct 26 15:25:55 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1789819392,\n'
' "io_kbytes" : 1747870,\n'
' "bw_bytes" : 255651962,\n'
' "bw" : 249660,\n'
' "iops" : 499319.240109,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3495734,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 23364,\n'
' "mean" : 152.689076,\n'
' "stddev" : '
'114.264283,\n'
' "N" : 3495734\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 6803,\n'
' "max" : 131567,\n'
' "mean" : '
'15002.125793,\n'
' "stddev" : '
'4088.111547,\n'
' "N" : 3495738,\n'
' "percentile" : {\n'
' "1.000000" : 9152,\n'
' "5.000000" : 12608,\n'
' "10.000000" : '
'12864,\n'
' "20.000000" : '
'13248,\n'
' "30.000000" : '
'13504,\n'
' "40.000000" : '
'13760,\n'
' "50.000000" : '
'14016,\n'
' "60.000000" : '
'14272,\n'
' "70.000000" : '
'14656,\n'
' "80.000000" : '
'15296,\n'
' "90.000000" : '
'18304,\n'
' "95.000000" : '
'21632,\n'
' "99.000000" : '
'33024,\n'
' "99.500000" : '
'34560,\n'
' "99.900000" : '
'51968,\n'
' "99.950000" : '
'67072,\n'
' "99.990000" : 78336\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6973,\n'
' "max" : 131697,\n'
' "mean" : '
'15154.814849,\n'
' "stddev" : '
'4085.047977,\n'
' "N" : 3495738\n'
' },\n'
' "bw_min" : 247874,\n'
' "bw_max" : 252834,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'249794.071429,\n'
' "bw_dev" : 1473.101410,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 495748,\n'
' "iops_max" : 505668,\n'
' "iops_mean" : '
'499588.285714,\n'
' "iops_stddev" : '
'2946.178762,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 30.042857,\n'
' "sys_cpu" : 69.928571,\n'
' "ctx" : 8,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.100000,\n'
' "4" : 72.377446,\n'
' "8" : 27.523175,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 98.713086,\n'
' "8" : 1.286914,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 1.834121,\n'
' "20" : 91.377834,\n'
' "50" : 6.677682,\n'
' "100" : 0.109390,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=10686: Wed '
'Oct 26 15:25:55 2022\n'
' read: IOPS=499k, BW=244MiB/s '
'(256MB/s)(1707MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=23364, avg=152.69, '
'stdev=114.26\n'
' clat (usec): min=6, max=131, '
'avg=15.00, stdev= 4.09\n'
' lat (usec): min=6, max=131, '
'avg=15.15, stdev= 4.09\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9152], '
'5.00th=[12608], 10.00th=[12864], '
'20.00th=[13248],\n'
' | 30.00th=[13504], '
'40.00th=[13760], '
'50.00th=[14016], '
'60.00th=[14272],\n'
' | 70.00th=[14656], '
'80.00th=[15296], '
'90.00th=[18304], '
'95.00th=[21632],\n'
' | 99.00th=[33024], '
'99.50th=[34560], '
'99.90th=[51968], '
'99.95th=[67072],\n'
' | 99.99th=[78336]\n'
' bw ( KiB/s): min=247874, '
'max=252834, per=100.00%, '
'avg=249794.07, stdev=1473.10, '
'samples=14\n'
' iops : min=495748, '
'max=505668, avg=499588.29, '
'stdev=2946.18, samples=14\n'
' lat (usec) : 10=1.83%, '
'20=91.38%, 50=6.68%, 100=0.11%, '
'250=0.01%\n'
' cpu : usr=30.04%, '
'sys=69.93%, ctx=8, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=72.4%, 8=27.5%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=98.7%, 8=1.3%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3495734,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=244MiB/s (256MB/s), '
'244MiB/s-244MiB/s '
'(256MB/s-256MB/s), io=1707MiB '
'(1790MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_336.output'),
'state': {'begin': 1666783665.3017976,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.79756760597229,
'end': 1666783666.0993652,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_336.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_336.state')},
'cmd_337': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_337.output'),
'state': {'begin': 1666783668.1847413,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.8187205791473389,
'end': 1666783669.0034618,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_337.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_337.state')},
'cmd_338': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_338.output'),
'state': {'begin': 1666783669.0102723,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.410768747329712,
'end': 1666783680.421041,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_338.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_338.state')},
'cmd_339': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790770,\n'
' "timestamp_ms" : '
'1666790770427,\n'
' "time" : "Wed Oct 26 15:26:10 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1791554560,\n'
' "io_kbytes" : 1749565,\n'
' "bw_bytes" : 255899808,\n'
' "bw" : 249902,\n'
' "iops" : 499803.313812,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3499123,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 24446,\n'
' "mean" : 156.179399,\n'
' "stddev" : '
'115.772836,\n'
' "N" : 3499123\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 6753,\n'
' "max" : 143268,\n'
' "mean" : '
'14972.242013,\n'
' "stddev" : '
'4066.042433,\n'
' "N" : 3499127,\n'
' "percentile" : {\n'
' "1.000000" : 9152,\n'
' "5.000000" : 12480,\n'
' "10.000000" : '
'12864,\n'
' "20.000000" : '
'13248,\n'
' "30.000000" : '
'13504,\n'
' "40.000000" : '
'13760,\n'
' "50.000000" : '
'14016,\n'
' "60.000000" : '
'14272,\n'
' "70.000000" : '
'14656,\n'
' "80.000000" : '
'15168,\n'
' "90.000000" : '
'18304,\n'
' "95.000000" : '
'21632,\n'
' "99.000000" : '
'33024,\n'
' "99.500000" : '
'34560,\n'
' "99.900000" : '
'50944,\n'
' "99.950000" : '
'67072,\n'
' "99.990000" : 78336\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6923,\n'
' "max" : 143469,\n'
' "mean" : '
'15128.421388,\n'
' "stddev" : '
'4063.050872,\n'
' "N" : 3499127\n'
' },\n'
' "bw_min" : 247329,\n'
' "bw_max" : 253134,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'250039.500000,\n'
' "bw_dev" : 1803.163577,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 494658,\n'
' "iops_max" : 506268,\n'
' "iops_mean" : '
'500079.142857,\n'
' "iops_stddev" : '
'3606.253582,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 29.642857,\n'
' "sys_cpu" : 70.328571,\n'
' "ctx" : 9,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.117515,\n'
' "4" : 72.370934,\n'
' "8" : 27.508121,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 98.752327,\n'
' "8" : 1.247673,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 1.845948,\n'
' "20" : 91.447028,\n'
' "50" : 6.605484,\n'
' "100" : 0.100740,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=10743: Wed '
'Oct 26 15:26:10 2022\n'
' read: IOPS=500k, BW=244MiB/s '
'(256MB/s)(1709MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=24446, avg=156.18, '
'stdev=115.77\n'
' clat (usec): min=6, max=143, '
'avg=14.97, stdev= 4.07\n'
' lat (usec): min=6, max=143, '
'avg=15.13, stdev= 4.06\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9152], '
'5.00th=[12480], 10.00th=[12864], '
'20.00th=[13248],\n'
' | 30.00th=[13504], '
'40.00th=[13760], '
'50.00th=[14016], '
'60.00th=[14272],\n'
' | 70.00th=[14656], '
'80.00th=[15168], '
'90.00th=[18304], '
'95.00th=[21632],\n'
' | 99.00th=[33024], '
'99.50th=[34560], '
'99.90th=[50944], '
'99.95th=[67072],\n'
' | 99.99th=[78336]\n'
' bw ( KiB/s): min=247329, '
'max=253134, per=100.00%, '
'avg=250039.50, stdev=1803.16, '
'samples=14\n'
' iops : min=494658, '
'max=506268, avg=500079.14, '
'stdev=3606.25, samples=14\n'
' lat (usec) : 10=1.85%, '
'20=91.45%, 50=6.61%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=29.64%, '
'sys=70.33%, ctx=9, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=72.4%, 8=27.5%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=98.8%, 8=1.2%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3499123,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=244MiB/s (256MB/s), '
'244MiB/s-244MiB/s '
'(256MB/s-256MB/s), io=1709MiB '
'(1792MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_339.output'),
'state': {'begin': 1666783680.4237885,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 1.214738368988037,
'end': 1666783681.638527,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_339.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_339.state')},
'cmd_34': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_34.output'),
'state': {'begin': 1666782202.960092,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7774829864501953,
'end': 1666782203.737575,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_34.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_34.state')},
'cmd_340': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_340.output'),
'state': {'begin': 1666783685.4833064,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 1.1595447063446045,
'end': 1666783686.642851,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_340.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_340.state')},
'cmd_341': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_341.output'),
'state': {'begin': 1666783686.650046,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.890761375427246,
'end': 1666783698.5408075,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_341.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_341.state')},
'cmd_342': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790788,\n'
' "timestamp_ms" : '
'1666790788540,\n'
' "time" : "Wed Oct 26 15:26:28 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1763398144,\n'
' "io_kbytes" : 1722068,\n'
' "bw_bytes" : 251878037,\n'
' "bw" : 245974,\n'
' "iops" : 491948.435938,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3444131,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 100,\n'
' "max" : 20659,\n'
' "mean" : 159.195923,\n'
' "stddev" : '
'134.215657,\n'
' "N" : 3444132\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 6412,\n'
' "max" : 125806,\n'
' "mean" : '
'15197.289828,\n'
' "stddev" : '
'4131.746276,\n'
' "N" : 3444137,\n'
' "percentile" : {\n'
' "1.000000" : 8896,\n'
' "5.000000" : 12608,\n'
' "10.000000" : '
'12992,\n'
' "20.000000" : '
'13376,\n'
' "30.000000" : '
'13632,\n'
' "40.000000" : '
'13888,\n'
' "50.000000" : '
'14144,\n'
' "60.000000" : '
'14528,\n'
' "70.000000" : '
'14912,\n'
' "80.000000" : '
'15552,\n'
' "90.000000" : '
'18816,\n'
' "95.000000" : '
'22400,\n'
' "99.000000" : '
'33024,\n'
' "99.500000" : '
'34560,\n'
' "99.900000" : '
'51456,\n'
' "99.950000" : '
'69120,\n'
' "99.990000" : 80384\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6612,\n'
' "max" : 126016,\n'
' "mean" : '
'15356.485787,\n'
' "stddev" : '
'4127.680316,\n'
' "N" : 3444137\n'
' },\n'
' "bw_min" : 244609,\n'
' "bw_max" : 248021,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'246103.357143,\n'
' "bw_dev" : 1102.233300,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 489218,\n'
' "iops_max" : 496042,\n'
' "iops_mean" : '
'492206.785714,\n'
' "iops_stddev" : '
'2204.746425,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 32.785714,\n'
' "sys_cpu" : 67.185714,\n'
' "ctx" : 12,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.129118,\n'
' "4" : 72.261014,\n'
' "8" : 27.606935,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 98.422303,\n'
' "8" : 1.577697,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 1.778126,\n'
' "20" : 90.700905,\n'
' "50" : 7.415281,\n'
' "100" : 0.104874,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=10800: Wed '
'Oct 26 15:26:28 2022\n'
' read: IOPS=492k, BW=240MiB/s '
'(252MB/s)(1682MiB/7001msec)\n'
' slat (nsec): min=100, '
'max=20659, avg=159.20, '
'stdev=134.22\n'
' clat (usec): min=6, max=125, '
'avg=15.20, stdev= 4.13\n'
' lat (usec): min=6, max=126, '
'avg=15.36, stdev= 4.13\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8896], '
'5.00th=[12608], 10.00th=[12992], '
'20.00th=[13376],\n'
' | 30.00th=[13632], '
'40.00th=[13888], '
'50.00th=[14144], '
'60.00th=[14528],\n'
' | 70.00th=[14912], '
'80.00th=[15552], '
'90.00th=[18816], '
'95.00th=[22400],\n'
' | 99.00th=[33024], '
'99.50th=[34560], '
'99.90th=[51456], '
'99.95th=[69120],\n'
' | 99.99th=[80384]\n'
' bw ( KiB/s): min=244609, '
'max=248021, per=100.00%, '
'avg=246103.36, stdev=1102.23, '
'samples=14\n'
' iops : min=489218, '
'max=496042, avg=492206.79, '
'stdev=2204.75, samples=14\n'
' lat (usec) : 10=1.78%, '
'20=90.70%, 50=7.42%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=32.79%, '
'sys=67.19%, ctx=12, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=72.3%, 8=27.6%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=98.4%, 8=1.6%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3444131,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=240MiB/s (252MB/s), '
'240MiB/s-240MiB/s '
'(252MB/s-252MB/s), io=1682MiB '
'(1763MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_342.output'),
'state': {'begin': 1666783698.5434163,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 1.2910566329956055,
'end': 1666783699.834473,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_342.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_342.state')},
'cmd_343': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_343.output'),
'state': {'begin': 1666783702.4286778,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 1.2081053256988525,
'end': 1666783703.6367831,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_343.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_343.state')},
'cmd_344': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_344.output'),
'state': {'begin': 1666783703.6387038,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 12.253406524658203,
'end': 1666783715.8921103,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_344.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_344.state')},
'cmd_345': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790805,\n'
' "timestamp_ms" : '
'1666790805897,\n'
' "time" : "Wed Oct 26 15:26:45 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1621752320,\n'
' "io_kbytes" : 1583742,\n'
' "bw_bytes" : 231645810,\n'
' "bw" : 226216,\n'
' "iops" : 452432.081131,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3167477,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1563,\n'
' "max" : 52558,\n'
' "mean" : 1732.091699,\n'
' "stddev" : '
'132.863734,\n'
' "N" : 3167478\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 250,\n'
' "max" : 126818,\n'
' "mean" : '
'15699.602769,\n'
' "stddev" : '
'2539.253016,\n'
' "N" : 3167484,\n'
' "percentile" : {\n'
' "1.000000" : 14912,\n'
' "5.000000" : 14912,\n'
' "10.000000" : '
'15040,\n'
' "20.000000" : '
'15040,\n'
' "30.000000" : '
'15168,\n'
' "40.000000" : '
'15168,\n'
' "50.000000" : '
'15296,\n'
' "60.000000" : '
'15296,\n'
' "70.000000" : '
'15296,\n'
' "80.000000" : '
'15424,\n'
' "90.000000" : '
'15808,\n'
' "95.000000" : '
'16192,\n'
' "99.000000" : '
'28288,\n'
' "99.500000" : '
'30592,\n'
' "99.900000" : '
'43776,\n'
' "99.950000" : '
'60672,\n'
' "99.990000" : 74240\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6412,\n'
' "max" : 128531,\n'
' "mean" : '
'17431.694401,\n'
' "stddev" : '
'2546.918829,\n'
' "N" : 3167484\n'
' },\n'
' "bw_min" : 224958,\n'
' "bw_max" : 227326,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'226257.285714,\n'
' "bw_dev" : 705.539339,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 449916,\n'
' "iops_max" : 454652,\n'
' "iops_mean" : '
'452514.642857,\n'
' "iops_stddev" : '
'1411.151936,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 30.385714,\n'
' "sys_cpu" : 69.600000,\n'
' "ctx" : 15,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999984,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.027404,\n'
' "20" : 96.891659,\n'
' "50" : 3.001000,\n'
' "100" : 0.078675,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=10858: Wed '
'Oct 26 15:26:45 2022\n'
' read: IOPS=452k, BW=221MiB/s '
'(232MB/s)(1547MiB/7001msec)\n'
' slat (nsec): min=1563, '
'max=52558, avg=1732.09, '
'stdev=132.86\n'
' clat (nsec): min=250, '
'max=126818, avg=15699.60, '
'stdev=2539.25\n'
' lat (usec): min=6, max=128, '
'avg=17.43, stdev= 2.55\n'
' clat percentiles (nsec):\n'
' | 1.00th=[14912], '
'5.00th=[14912], 10.00th=[15040], '
'20.00th=[15040],\n'
' | 30.00th=[15168], '
'40.00th=[15168], '
'50.00th=[15296], '
'60.00th=[15296],\n'
' | 70.00th=[15296], '
'80.00th=[15424], '
'90.00th=[15808], '
'95.00th=[16192],\n'
' | 99.00th=[28288], '
'99.50th=[30592], '
'99.90th=[43776], '
'99.95th=[60672],\n'
' | 99.99th=[74240]\n'
' bw ( KiB/s): min=224958, '
'max=227326, per=100.00%, '
'avg=226257.29, stdev=705.54, '
'samples=14\n'
' iops : min=449916, '
'max=454652, avg=452514.64, '
'stdev=1411.15, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 10=0.03%, '
'20=96.89%, 50=3.00%, 100=0.08%, '
'250=0.01%\n'
' cpu : usr=30.39%, '
'sys=69.60%, ctx=15, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3167477,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=221MiB/s (232MB/s), '
'221MiB/s-221MiB/s '
'(232MB/s-232MB/s), io=1547MiB '
'(1622MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_345.output'),
'state': {'begin': 1666783715.8946543,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 1.7201216220855713,
'end': 1666783717.614776,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_345.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_345.state')},
'cmd_346': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_346.output'),
'state': {'begin': 1666783720.8414304,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 1.889601469039917,
'end': 1666783722.731032,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_346.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_346.state')},
'cmd_347': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_347.output'),
'state': {'begin': 1666783722.7332733,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.909066915512085,
'end': 1666783734.6423402,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_347.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_347.state')},
'cmd_348': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790824,\n'
' "timestamp_ms" : '
'1666790824648,\n'
' "time" : "Wed Oct 26 15:27:04 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1612016128,\n'
' "io_kbytes" : 1574234,\n'
' "bw_bytes" : 230255124,\n'
' "bw" : 224858,\n'
' "iops" : 449715.897729,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3148461,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1573,\n'
' "max" : 40045,\n'
' "mean" : 1761.557793,\n'
' "stddev" : '
'123.973748,\n'
' "N" : 3148462\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 250,\n'
' "max" : 120826,\n'
' "mean" : '
'15794.247972,\n'
' "stddev" : '
'2581.682894,\n'
' "N" : 3148468,\n'
' "percentile" : {\n'
' "1.000000" : 14912,\n'
' "5.000000" : 15040,\n'
' "10.000000" : '
'15040,\n'
' "20.000000" : '
'15168,\n'
' "30.000000" : '
'15168,\n'
' "40.000000" : '
'15296,\n'
' "50.000000" : '
'15296,\n'
' "60.000000" : '
'15424,\n'
' "70.000000" : '
'15424,\n'
' "80.000000" : '
'15808,\n'
' "90.000000" : '
'16064,\n'
' "95.000000" : '
'16192,\n'
' "99.000000" : '
'28032,\n'
' "99.500000" : '
'30592,\n'
' "99.900000" : '
'49408,\n'
' "99.950000" : '
'64768,\n'
' "99.990000" : 75264\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5831,\n'
' "max" : 122500,\n'
' "mean" : '
'17555.805718,\n'
' "stddev" : '
'2589.532167,\n'
' "N" : 3148468\n'
' },\n'
' "bw_min" : 224054,\n'
' "bw_max" : 227215,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'224976.142857,\n'
' "bw_dev" : 840.788902,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 448108,\n'
' "iops_max" : 454430,\n'
' "iops_mean" : '
'449952.428571,\n'
' "iops_stddev" : '
'1681.479647,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 31.585714,\n'
' "sys_cpu" : 68.400000,\n'
' "ctx" : 11,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999984,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.032619,\n'
' "20" : 96.921004,\n'
' "50" : 2.948266,\n'
' "100" : 0.097254,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=10915: Wed '
'Oct 26 15:27:04 2022\n'
' read: IOPS=450k, BW=220MiB/s '
'(230MB/s)(1537MiB/7001msec)\n'
' slat (nsec): min=1573, '
'max=40045, avg=1761.56, '
'stdev=123.97\n'
' clat (nsec): min=250, '
'max=120826, avg=15794.25, '
'stdev=2581.68\n'
' lat (usec): min=5, max=122, '
'avg=17.56, stdev= 2.59\n'
' clat percentiles (nsec):\n'
' | 1.00th=[14912], '
'5.00th=[15040], 10.00th=[15040], '
'20.00th=[15168],\n'
' | 30.00th=[15168], '
'40.00th=[15296], '
'50.00th=[15296], '
'60.00th=[15424],\n'
' | 70.00th=[15424], '
'80.00th=[15808], '
'90.00th=[16064], '
'95.00th=[16192],\n'
' | 99.00th=[28032], '
'99.50th=[30592], '
'99.90th=[49408], '
'99.95th=[64768],\n'
' | 99.99th=[75264]\n'
' bw ( KiB/s): min=224054, '
'max=227215, per=100.00%, '
'avg=224976.14, stdev=840.79, '
'samples=14\n'
' iops : min=448108, '
'max=454430, avg=449952.43, '
'stdev=1681.48, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 10=0.03%, '
'20=96.92%, 50=2.95%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=31.59%, '
'sys=68.40%, ctx=11, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3148461,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=220MiB/s (230MB/s), '
'220MiB/s-220MiB/s '
'(230MB/s-230MB/s), io=1537MiB '
'(1612MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_348.output'),
'state': {'begin': 1666783734.6453257,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 1.0389409065246582,
'end': 1666783735.6842666,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_348.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_348.state')},
'cmd_349': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_349.output'),
'state': {'begin': 1666783738.3035614,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 1.1200616359710693,
'end': 1666783739.423623,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_349.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_349.state')},
'cmd_35': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_35.output'),
'state': {'begin': 1666782203.7398446,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.05388355255127,
'end': 1666782214.793728,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_35.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_35.state')},
'cmd_350': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_350.output'),
'state': {'begin': 1666783739.425886,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.944929599761963,
'end': 1666783751.3708155,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_350.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_350.state')},
'cmd_351': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790841,\n'
' "timestamp_ms" : '
'1666790841375,\n'
' "time" : "Wed Oct 26 15:27:21 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1616416768,\n'
' "io_kbytes" : 1578532,\n'
' "bw_bytes" : 230883697,\n'
' "bw" : 225472,\n'
' "iops" : 450943.579489,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3157056,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1572,\n'
' "max" : 17793,\n'
' "mean" : 1750.871442,\n'
' "stddev" : '
'124.108060,\n'
' "N" : 3157057\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 260,\n'
' "max" : 128782,\n'
' "mean" : '
'15750.550326,\n'
' "stddev" : '
'2540.179660,\n'
' "N" : 3157063,\n'
' "percentile" : {\n'
' "1.000000" : 14912,\n'
' "5.000000" : 15040,\n'
' "10.000000" : '
'15040,\n'
' "20.000000" : '
'15168,\n'
' "30.000000" : '
'15168,\n'
' "40.000000" : '
'15168,\n'
' "50.000000" : '
'15296,\n'
' "60.000000" : '
'15296,\n'
' "70.000000" : '
'15424,\n'
' "80.000000" : '
'15808,\n'
' "90.000000" : '
'15936,\n'
' "95.000000" : '
'16192,\n'
' "99.000000" : '
'28288,\n'
' "99.500000" : '
'30592,\n'
' "99.900000" : '
'44288,\n'
' "99.950000" : '
'63232,\n'
' "99.990000" : 74240\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6081,\n'
' "max" : 130635,\n'
' "mean" : '
'17501.421643,\n'
' "stddev" : '
'2547.987942,\n'
' "N" : 3157063\n'
' },\n'
' "bw_min" : 224516,\n'
' "bw_max" : 226843,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'225679.230769,\n'
' "bw_dev" : 917.421582,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 449032,\n'
' "iops_max" : 453686,\n'
' "iops_mean" : '
'451358.615385,\n'
' "iops_stddev" : '
'1835.033630,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 31.128571,\n'
' "sys_cpu" : 68.857143,\n'
' "ctx" : 8,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999984,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.031517,\n'
' "20" : 96.933409,\n'
' "50" : 2.949868,\n'
' "100" : 0.084382,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=10974: Wed '
'Oct 26 15:27:21 2022\n'
' read: IOPS=451k, BW=220MiB/s '
'(231MB/s)(1542MiB/7001msec)\n'
' slat (nsec): min=1572, '
'max=17793, avg=1750.87, '
'stdev=124.11\n'
' clat (nsec): min=260, '
'max=128782, avg=15750.55, '
'stdev=2540.18\n'
' lat (usec): min=6, max=130, '
'avg=17.50, stdev= 2.55\n'
' clat percentiles (nsec):\n'
' | 1.00th=[14912], '
'5.00th=[15040], 10.00th=[15040], '
'20.00th=[15168],\n'
' | 30.00th=[15168], '
'40.00th=[15168], '
'50.00th=[15296], '
'60.00th=[15296],\n'
' | 70.00th=[15424], '
'80.00th=[15808], '
'90.00th=[15936], '
'95.00th=[16192],\n'
' | 99.00th=[28288], '
'99.50th=[30592], '
'99.90th=[44288], '
'99.95th=[63232],\n'
' | 99.99th=[74240]\n'
' bw ( KiB/s): min=224516, '
'max=226843, per=100.00%, '
'avg=225679.23, stdev=917.42, '
'samples=13\n'
' iops : min=449032, '
'max=453686, avg=451358.62, '
'stdev=1835.03, samples=13\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.03%, 20=96.93%, 50=2.95%, '
'100=0.08%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=31.13%, '
'sys=68.86%, ctx=8, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3157056,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=220MiB/s (231MB/s), '
'220MiB/s-220MiB/s '
'(231MB/s-231MB/s), io=1542MiB '
'(1616MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_351.output'),
'state': {'begin': 1666783751.3734238,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 1.1073675155639648,
'end': 1666783752.4807913,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_351.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_351.state')},
'cmd_352': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_352.output'),
'state': {'begin': 1666783753.932052,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 1.4276745319366455,
'end': 1666783755.3597264,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_352.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_352.state')},
'cmd_353': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_353.output'),
'state': {'begin': 1666783755.3624756,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.359984636306763,
'end': 1666783766.7224603,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_353.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_353.state')},
'cmd_354': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790856,\n'
' "timestamp_ms" : '
'1666790856790,\n'
' "time" : "Wed Oct 26 15:27:36 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1419834368,\n'
' "io_kbytes" : 1386557,\n'
' "bw_bytes" : 202804509,\n'
' "bw" : 198051,\n'
' "iops" : 396101.414084,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2773106,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1393,\n'
' "max" : 70382,\n'
' "mean" : 2032.974013,\n'
' "stddev" : '
'315.234496,\n'
' "N" : 2773107\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4167,\n'
' "max" : 134191,\n'
' "mean" : '
'17867.042998,\n'
' "stddev" : '
'3010.820945,\n'
' "N" : 2773113,\n'
' "percentile" : {\n'
' "1.000000" : 13376,\n'
' "5.000000" : 15424,\n'
' "10.000000" : '
'16768,\n'
' "20.000000" : '
'17024,\n'
' "30.000000" : '
'17280,\n'
' "40.000000" : '
'17536,\n'
' "50.000000" : '
'17536,\n'
' "60.000000" : '
'17536,\n'
' "70.000000" : '
'18048,\n'
' "80.000000" : '
'18048,\n'
' "90.000000" : '
'18304,\n'
' "95.000000" : '
'19584,\n'
' "99.000000" : '
'31616,\n'
' "99.500000" : '
'34560,\n'
' "99.900000" : '
'50432,\n'
' "99.950000" : '
'64256,\n'
' "99.990000" : 77312\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6372,\n'
' "max" : 135744,\n'
' "mean" : '
'19900.015943,\n'
' "stddev" : '
'2965.963506,\n'
' "N" : 2773113\n'
' },\n'
' "bw_min" : 197423,\n'
' "bw_max" : 198917,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'198149.500000,\n'
' "bw_dev" : 471.310897,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 394847,\n'
' "iops_max" : 397834,\n'
' "iops_mean" : '
'396299.142857,\n'
' "iops_stddev" : '
'942.487532,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 20.971429,\n'
' "sys_cpu" : 76.928571,\n'
' "ctx" : 16494,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999982,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.102989,\n'
' "20" : 95.387482,\n'
' "50" : 4.409135,\n'
' "100" : 0.099780,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=11022: Wed '
'Oct 26 15:27:36 2022\n'
' read: IOPS=396k, BW=193MiB/s '
'(203MB/s)(1354MiB/7001msec)\n'
' slat (nsec): min=1393, '
'max=70382, avg=2032.97, '
'stdev=315.23\n'
' clat (usec): min=4, max=134, '
'avg=17.87, stdev= 3.01\n'
' lat (usec): min=6, max=135, '
'avg=19.90, stdev= 2.97\n'
' clat percentiles (nsec):\n'
' | 1.00th=[13376], '
'5.00th=[15424], 10.00th=[16768], '
'20.00th=[17024],\n'
' | 30.00th=[17280], '
'40.00th=[17536], '
'50.00th=[17536], '
'60.00th=[17536],\n'
' | 70.00th=[18048], '
'80.00th=[18048], '
'90.00th=[18304], '
'95.00th=[19584],\n'
' | 99.00th=[31616], '
'99.50th=[34560], '
'99.90th=[50432], '
'99.95th=[64256],\n'
' | 99.99th=[77312]\n'
' bw ( KiB/s): min=197423, '
'max=198917, per=100.00%, '
'avg=198149.50, stdev=471.31, '
'samples=14\n'
' iops : min=394847, '
'max=397834, avg=396299.14, '
'stdev=942.49, samples=14\n'
' lat (usec) : 10=0.10%, '
'20=95.39%, 50=4.41%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=20.97%, '
'sys=76.93%, ctx=16494, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2773106,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=193MiB/s (203MB/s), '
'193MiB/s-193MiB/s '
'(203MB/s-203MB/s), io=1354MiB '
'(1420MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_354.output'),
'state': {'begin': 1666783766.7253551,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 1.3404881954193115,
'end': 1666783768.0658433,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_354.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_354.state')},
'cmd_355': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_355.output'),
'state': {'begin': 1666783769.5046735,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 1.3336870670318604,
'end': 1666783770.8383605,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_355.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_355.state')},
'cmd_356': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_356.output'),
'state': {'begin': 1666783770.8401787,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 12.943699359893799,
'end': 1666783783.783878,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_356.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_356.state')},
'cmd_357': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790873,\n'
' "timestamp_ms" : '
'1666790873848,\n'
' "time" : "Wed Oct 26 15:27:53 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1414161408,\n'
' "io_kbytes" : 1381017,\n'
' "bw_bytes" : 201994201,\n'
' "bw" : 197259,\n'
' "iops" : 394518.925868,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2762027,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1392,\n'
' "max" : 30357,\n'
' "mean" : 2037.367052,\n'
' "stddev" : '
'294.648162,\n'
' "N" : 2762027\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4098,\n'
' "max" : 135283,\n'
' "mean" : '
'17936.435806,\n'
' "stddev" : '
'3001.136611,\n'
' "N" : 2762034,\n'
' "percentile" : {\n'
' "1.000000" : 13376,\n'
' "5.000000" : 15552,\n'
' "10.000000" : '
'16768,\n'
' "20.000000" : '
'17280,\n'
' "30.000000" : '
'17536,\n'
' "40.000000" : '
'17536,\n'
' "50.000000" : '
'17536,\n'
' "60.000000" : '
'17536,\n'
' "70.000000" : '
'18048,\n'
' "80.000000" : '
'18304,\n'
' "90.000000" : '
'18304,\n'
' "95.000000" : '
'19584,\n'
' "99.000000" : '
'31616,\n'
' "99.500000" : '
'35072,\n'
' "99.900000" : '
'51456,\n'
' "99.950000" : '
'66048,\n'
' "99.990000" : 76288\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7033,\n'
' "max" : 136786,\n'
' "mean" : '
'19973.801927,\n'
' "stddev" : '
'2955.591710,\n'
' "N" : 2762034\n'
' },\n'
' "bw_min" : 196032,\n'
' "bw_max" : 198923,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'197370.428571,\n'
' "bw_dev" : 916.608608,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 392064,\n'
' "iops_max" : 397846,\n'
' "iops_mean" : '
'394741.000000,\n'
' "iops_stddev" : '
'1833.052687,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 20.842857,\n'
' "sys_cpu" : 77.085714,\n'
' "ctx" : 16725,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999982,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.081208,\n'
' "20" : 95.414129,\n'
' "50" : 4.391956,\n'
' "100" : 0.112671,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=11070: Wed '
'Oct 26 15:27:53 2022\n'
' read: IOPS=395k, BW=193MiB/s '
'(202MB/s)(1349MiB/7001msec)\n'
' slat (nsec): min=1392, '
'max=30357, avg=2037.37, '
'stdev=294.65\n'
' clat (usec): min=4, max=135, '
'avg=17.94, stdev= 3.00\n'
' lat (usec): min=7, max=136, '
'avg=19.97, stdev= 2.96\n'
' clat percentiles (nsec):\n'
' | 1.00th=[13376], '
'5.00th=[15552], 10.00th=[16768], '
'20.00th=[17280],\n'
' | 30.00th=[17536], '
'40.00th=[17536], '
'50.00th=[17536], '
'60.00th=[17536],\n'
' | 70.00th=[18048], '
'80.00th=[18304], '
'90.00th=[18304], '
'95.00th=[19584],\n'
' | 99.00th=[31616], '
'99.50th=[35072], '
'99.90th=[51456], '
'99.95th=[66048],\n'
' | 99.99th=[76288]\n'
' bw ( KiB/s): min=196032, '
'max=198923, per=100.00%, '
'avg=197370.43, stdev=916.61, '
'samples=14\n'
' iops : min=392064, '
'max=397846, avg=394741.00, '
'stdev=1833.05, samples=14\n'
' lat (usec) : 10=0.08%, '
'20=95.41%, 50=4.39%, 100=0.11%, '
'250=0.01%\n'
' cpu : usr=20.84%, '
'sys=77.09%, ctx=16725, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2762027,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=193MiB/s (202MB/s), '
'193MiB/s-193MiB/s '
'(202MB/s-202MB/s), io=1349MiB '
'(1414MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_357.output'),
'state': {'begin': 1666783783.7866464,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 1.2279436588287354,
'end': 1666783785.01459,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_357.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_357.state')},
'cmd_358': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_io_uring-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_358.output'),
'state': {'begin': 1666783788.3961504,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 1.418452501296997,
'end': 1666783789.8146029,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_358.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_358.state')},
'cmd_359': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_359.output'),
'state': {'begin': 1666783789.81669,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.817346572875977,
'end': 1666783801.6340365,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_359.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_359.state')},
'cmd_36': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789304,\n'
' "timestamp_ms" : '
'1666789304886,\n'
' "time" : "Wed Oct 26 15:01:44 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 393098240,\n'
' "io_kbytes" : 383885,\n'
' "bw_bytes" : 56148870,\n'
' "bw" : 54832,\n'
' "iops" : 109665.762034,\n'
' "runtime" : 7001,\n'
' "total_ios" : 767770,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1493,\n'
' "max" : 6332,\n'
' "mean" : 1588.558791,\n'
' "stddev" : 74.080772,\n'
' "N" : 767770\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2945,\n'
' "max" : 114725,\n'
' "mean" : 7224.469301,\n'
' "stddev" : 999.991459,\n'
' "N" : 767770,\n'
' "percentile" : {\n'
' "1.000000" : 6752,\n'
' "5.000000" : 6944,\n'
' "10.000000" : 6944,\n'
' "20.000000" : 7008,\n'
' "30.000000" : 7008,\n'
' "40.000000" : 7072,\n'
' "50.000000" : 7136,\n'
' "60.000000" : 7136,\n'
' "70.000000" : 7200,\n'
' "80.000000" : 7200,\n'
' "90.000000" : 7328,\n'
' "95.000000" : 7456,\n'
' "99.000000" : 10432,\n'
' "99.500000" : 13632,\n'
' "99.900000" : 15552,\n'
' "99.950000" : 15808,\n'
' "99.990000" : 59136\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7434,\n'
' "max" : 116268,\n'
' "mean" : 8813.028092,\n'
' "stddev" : '
'1004.849320,\n'
' "N" : 767770\n'
' },\n'
' "bw_min" : 54684,\n'
' "bw_max" : 55007,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'54870.642857,\n'
' "bw_dev" : 100.171090,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 109368,\n'
' "iops_max" : 110014,\n'
' "iops_mean" : '
'109741.500000,\n'
' "iops_stddev" : '
'200.451701,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 7.971429,\n'
' "sys_cpu" : 35.742857,\n'
' "ctx" : 767772,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 98.707556,\n'
' "20" : 1.272517,\n'
' "50" : 0.010000,\n'
' "100" : 0.014588,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 1100470,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 5516,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 5516,\n'
' "util" : 99.108381\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=5483: Wed '
'Oct 26 15:01:44 2022\n'
' read: IOPS=110k, BW=53.5MiB/s '
'(56.1MB/s)(375MiB/7001msec)\n'
' slat (nsec): min=1493, '
'max=6332, avg=1588.56, '
'stdev=74.08\n'
' clat (nsec): min=2945, '
'max=114725, avg=7224.47, '
'stdev=999.99\n'
' lat (usec): min=7, max=116, '
'avg= 8.81, stdev= 1.00\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6752], 5.00th=[ '
'6944], 10.00th=[ 6944], 20.00th=[ '
'7008],\n'
' | 30.00th=[ 7008], 40.00th=[ '
'7072], 50.00th=[ 7136], 60.00th=[ '
'7136],\n'
' | 70.00th=[ 7200], 80.00th=[ '
'7200], 90.00th=[ 7328], 95.00th=[ '
'7456],\n'
' | 99.00th=[10432], '
'99.50th=[13632], 99.90th=[15552], '
'99.95th=[15808],\n'
' | 99.99th=[59136]\n'
' bw ( KiB/s): min=54684, '
'max=55007, per=100.00%, '
'avg=54870.64, stdev=100.17, '
'samples=14\n'
' iops : min=109368, '
'max=110014, avg=109741.50, '
'stdev=200.45, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=98.71%, 20=1.27%, 50=0.01%, '
'100=0.01%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=7.97%, '
'sys=35.74%, ctx=767772, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=767770,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=53.5MiB/s (56.1MB/s), '
'53.5MiB/s-53.5MiB/s '
'(56.1MB/s-56.1MB/s), io=375MiB '
'(393MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=1100470/0, '
'merge=0/0, ticks=5516/0, '
'in_queue=5516, util=99.11%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_36.output'),
'state': {'begin': 1666782214.7961264,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.8309309482574463,
'end': 1666782215.6270573,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_36.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_36.state')},
'cmd_360': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790891,\n'
' "timestamp_ms" : '
'1666790891701,\n'
' "time" : "Wed Oct 26 15:28:11 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1422672384,\n'
' "io_kbytes" : 1389328,\n'
' "bw_bytes" : 203209882,\n'
' "bw" : 198447,\n'
' "iops" : 396893.158120,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2778649,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1402,\n'
' "max" : 29996,\n'
' "mean" : 2024.455626,\n'
' "stddev" : '
'305.377303,\n'
' "N" : 2778650\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2254,\n'
' "max" : 131687,\n'
' "mean" : '
'17831.837688,\n'
' "stddev" : '
'3036.634295,\n'
' "N" : 2778656,\n'
' "percentile" : {\n'
' "1.000000" : 13248,\n'
' "5.000000" : 15168,\n'
' "10.000000" : '
'16512,\n'
' "20.000000" : '
'17024,\n'
' "30.000000" : '
'17280,\n'
' "40.000000" : '
'17536,\n'
' "50.000000" : '
'17536,\n'
' "60.000000" : '
'17536,\n'
' "70.000000" : '
'17536,\n'
' "80.000000" : '
'18048,\n'
' "90.000000" : '
'18304,\n'
' "95.000000" : '
'19584,\n'
' "99.000000" : '
'31872,\n'
' "99.500000" : '
'35072,\n'
' "99.900000" : '
'50944,\n'
' "99.950000" : '
'64768,\n'
' "99.990000" : 77312\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6422,\n'
' "max" : 133160,\n'
' "mean" : '
'19856.292749,\n'
' "stddev" : '
'2987.817199,\n'
' "N" : 2778656\n'
' },\n'
' "bw_min" : 197185,\n'
' "bw_max" : 199877,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'198482.571429,\n'
' "bw_dev" : 752.010404,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 394370,\n'
' "iops_max" : 399754,\n'
' "iops_mean" : '
'396965.142857,\n'
' "iops_stddev" : '
'1504.020809,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 22.071429,\n'
' "sys_cpu" : 75.800000,\n'
' "ctx" : 16640,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999982,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.090620,\n'
' "20" : 95.460888,\n'
' "50" : 4.343226,\n'
' "100" : 0.104331,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=11118: Wed '
'Oct 26 15:28:11 2022\n'
' read: IOPS=397k, BW=194MiB/s '
'(203MB/s)(1357MiB/7001msec)\n'
' slat (nsec): min=1402, '
'max=29996, avg=2024.46, '
'stdev=305.38\n'
' clat (usec): min=2, max=131, '
'avg=17.83, stdev= 3.04\n'
' lat (usec): min=6, max=133, '
'avg=19.86, stdev= 2.99\n'
' clat percentiles (nsec):\n'
' | 1.00th=[13248], '
'5.00th=[15168], 10.00th=[16512], '
'20.00th=[17024],\n'
' | 30.00th=[17280], '
'40.00th=[17536], '
'50.00th=[17536], '
'60.00th=[17536],\n'
' | 70.00th=[17536], '
'80.00th=[18048], '
'90.00th=[18304], '
'95.00th=[19584],\n'
' | 99.00th=[31872], '
'99.50th=[35072], '
'99.90th=[50944], '
'99.95th=[64768],\n'
' | 99.99th=[77312]\n'
' bw ( KiB/s): min=197185, '
'max=199877, per=100.00%, '
'avg=198482.57, stdev=752.01, '
'samples=14\n'
' iops : min=394370, '
'max=399754, avg=396965.14, '
'stdev=1504.02, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.09%, 20=95.46%, 50=4.34%, '
'100=0.10%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=22.07%, '
'sys=75.80%, ctx=16640, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2778649,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=194MiB/s (203MB/s), '
'194MiB/s-194MiB/s '
'(203MB/s-203MB/s), io=1357MiB '
'(1423MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_360.output'),
'state': {'begin': 1666783801.637204,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 1.2167508602142334,
'end': 1666783802.8539548,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_360.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_360.state')},
'cmd_361': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_libaio-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_361.output'),
'state': {'begin': 1666783804.3302684,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 1.5104670524597168,
'end': 1666783805.8407354,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_361.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_361.state')},
'cmd_362': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_362.output'),
'state': {'begin': 1666783805.8427002,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.908896923065186,
'end': 1666783817.7515972,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_362.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_362.state')},
'cmd_363': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790907,\n'
' "timestamp_ms" : '
'1666790907820,\n'
' "time" : "Wed Oct 26 15:28:27 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1573419008,\n'
' "io_kbytes" : 1536542,\n'
' "bw_bytes" : 224742037,\n'
' "bw" : 219474,\n'
' "iops" : 438948.293101,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3073077,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1442,\n'
' "max" : 20178,\n'
' "mean" : 1602.113093,\n'
' "stddev" : '
'122.876606,\n'
' "N" : 3073077\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4669,\n'
' "max" : 139171,\n'
' "mean" : '
'16380.314330,\n'
' "stddev" : '
'3042.824860,\n'
' "N" : 3073084,\n'
' "percentile" : {\n'
' "1.000000" : 13888,\n'
' "5.000000" : 15552,\n'
' "10.000000" : '
'15680,\n'
' "20.000000" : '
'15680,\n'
' "30.000000" : '
'15680,\n'
' "40.000000" : '
'15808,\n'
' "50.000000" : '
'15808,\n'
' "60.000000" : '
'15808,\n'
' "70.000000" : '
'15936,\n'
' "80.000000" : '
'16320,\n'
' "90.000000" : '
'16512,\n'
' "95.000000" : '
'17536,\n'
' "99.000000" : '
'31616,\n'
' "99.500000" : '
'35584,\n'
' "99.900000" : '
'51456,\n'
' "99.950000" : '
'66048,\n'
' "99.990000" : 78336\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6172,\n'
' "max" : 140724,\n'
' "mean" : '
'17982.427350,\n'
' "stddev" : '
'3044.762520,\n'
' "N" : 3073084\n'
' },\n'
' "bw_min" : 218423,\n'
' "bw_max" : 220928,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'219536.928571,\n'
' "bw_dev" : 658.497411,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 436846,\n'
' "iops_max" : 441857,\n'
' "iops_mean" : '
'439073.928571,\n'
' "iops_stddev" : '
'1317.157339,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 30.700000,\n'
' "sys_cpu" : 67.314286,\n'
' "ctx" : 18155,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999967,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.426283,\n'
' "20" : 95.925061,\n'
' "50" : 3.544721,\n'
' "100" : 0.103024,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 4442260,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 25258,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 25258,\n'
' "util" : 99.069123\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=11167: Wed '
'Oct 26 15:28:27 2022\n'
' read: IOPS=439k, BW=214MiB/s '
'(225MB/s)(1501MiB/7001msec)\n'
' slat (nsec): min=1442, '
'max=20178, avg=1602.11, '
'stdev=122.88\n'
' clat (usec): min=4, max=139, '
'avg=16.38, stdev= 3.04\n'
' lat (usec): min=6, max=140, '
'avg=17.98, stdev= 3.04\n'
' clat percentiles (nsec):\n'
' | 1.00th=[13888], '
'5.00th=[15552], 10.00th=[15680], '
'20.00th=[15680],\n'
' | 30.00th=[15680], '
'40.00th=[15808], '
'50.00th=[15808], '
'60.00th=[15808],\n'
' | 70.00th=[15936], '
'80.00th=[16320], '
'90.00th=[16512], '
'95.00th=[17536],\n'
' | 99.00th=[31616], '
'99.50th=[35584], '
'99.90th=[51456], '
'99.95th=[66048],\n'
' | 99.99th=[78336]\n'
' bw ( KiB/s): min=218423, '
'max=220928, per=100.00%, '
'avg=219536.93, stdev=658.50, '
'samples=14\n'
' iops : min=436846, '
'max=441857, avg=439073.93, '
'stdev=1317.16, samples=14\n'
' lat (usec) : 10=0.43%, '
'20=95.93%, 50=3.54%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=30.70%, '
'sys=67.31%, ctx=18155, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3073077,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=214MiB/s (225MB/s), '
'214MiB/s-214MiB/s '
'(225MB/s-225MB/s), io=1501MiB '
'(1573MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=4442260/0, '
'merge=0/0, ticks=25258/0, '
'in_queue=25258, util=99.07%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_363.output'),
'state': {'begin': 1666783817.7535267,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 1.55983567237854,
'end': 1666783819.3133624,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_363.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_363.state')},
'cmd_364': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_libaio-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_364.output'),
'state': {'begin': 1666783820.844429,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 1.3285760879516602,
'end': 1666783822.173005,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_364.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_364.state')},
'cmd_365': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_365.output'),
'state': {'begin': 1666783822.1749372,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.932475805282593,
'end': 1666783834.107413,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_365.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_365.state')},
'cmd_366': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790924,\n'
' "timestamp_ms" : '
'1666790924175,\n'
' "time" : "Wed Oct 26 15:28:44 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1567867904,\n'
' "io_kbytes" : 1531121,\n'
' "bw_bytes" : 223949136,\n'
' "bw" : 218700,\n'
' "iops" : 437399.514355,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3062234,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1442,\n'
' "max" : 80601,\n'
' "mean" : 1606.787758,\n'
' "stddev" : '
'104.510126,\n'
' "N" : 3062235\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4960,\n'
' "max" : 135585,\n'
' "mean" : '
'16440.666159,\n'
' "stddev" : '
'2924.034150,\n'
' "N" : 3062241,\n'
' "percentile" : {\n'
' "1.000000" : 13760,\n'
' "5.000000" : 15680,\n'
' "10.000000" : '
'15680,\n'
' "20.000000" : '
'15808,\n'
' "30.000000" : '
'15808,\n'
' "40.000000" : '
'15808,\n'
' "50.000000" : '
'15936,\n'
' "60.000000" : '
'15936,\n'
' "70.000000" : '
'16064,\n'
' "80.000000" : '
'16512,\n'
' "90.000000" : '
'16512,\n'
' "95.000000" : '
'17536,\n'
' "99.000000" : '
'31104,\n'
' "99.500000" : '
'35072,\n'
' "99.900000" : '
'48896,\n'
' "99.950000" : '
'63744,\n'
' "99.990000" : 78336\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6483,\n'
' "max" : 137268,\n'
' "mean" : '
'18047.453991,\n'
' "stddev" : '
'2925.765710,\n'
' "N" : 3062241\n'
' },\n'
' "bw_min" : 218165,\n'
' "bw_max" : 219778,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'218823.571429,\n'
' "bw_dev" : 414.416868,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 436330,\n'
' "iops_max" : 439557,\n'
' "iops_mean" : '
'437647.285714,\n'
' "iops_stddev" : '
'829.026624,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 31.600000,\n'
' "sys_cpu" : 66.528571,\n'
' "ctx" : 17763,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999967,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.421882,\n'
' "20" : 96.022054,\n'
' "50" : 3.461231,\n'
' "100" : 0.094180,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 4419891,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 25004,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 25004,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=11213: Wed '
'Oct 26 15:28:44 2022\n'
' read: IOPS=437k, BW=214MiB/s '
'(224MB/s)(1495MiB/7001msec)\n'
' slat (nsec): min=1442, '
'max=80601, avg=1606.79, '
'stdev=104.51\n'
' clat (usec): min=4, max=135, '
'avg=16.44, stdev= 2.92\n'
' lat (usec): min=6, max=137, '
'avg=18.05, stdev= 2.93\n'
' clat percentiles (nsec):\n'
' | 1.00th=[13760], '
'5.00th=[15680], 10.00th=[15680], '
'20.00th=[15808],\n'
' | 30.00th=[15808], '
'40.00th=[15808], '
'50.00th=[15936], '
'60.00th=[15936],\n'
' | 70.00th=[16064], '
'80.00th=[16512], '
'90.00th=[16512], '
'95.00th=[17536],\n'
' | 99.00th=[31104], '
'99.50th=[35072], '
'99.90th=[48896], '
'99.95th=[63744],\n'
' | 99.99th=[78336]\n'
' bw ( KiB/s): min=218165, '
'max=219778, per=100.00%, '
'avg=218823.57, stdev=414.42, '
'samples=14\n'
' iops : min=436330, '
'max=439557, avg=437647.29, '
'stdev=829.03, samples=14\n'
' lat (usec) : 10=0.42%, '
'20=96.02%, 50=3.46%, 100=0.09%, '
'250=0.01%\n'
' cpu : usr=31.60%, '
'sys=66.53%, ctx=17763, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3062234,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=214MiB/s (224MB/s), '
'214MiB/s-214MiB/s '
'(224MB/s-224MB/s), io=1495MiB '
'(1568MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=4419891/0, '
'merge=0/0, ticks=25004/0, '
'in_queue=25004, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_366.output'),
'state': {'begin': 1666783834.1101973,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.9997060298919678,
'end': 1666783835.1099033,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_366.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_366.state')},
'cmd_367': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_libaio-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_367.output'),
'state': {'begin': 1666783836.7236187,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 1.7480182647705078,
'end': 1666783838.471637,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_367.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_367.state')},
'cmd_368': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_368.output'),
'state': {'begin': 1666783838.4743297,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 13.069213151931763,
'end': 1666783851.5435429,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_368.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_368.state')},
'cmd_369': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790941,\n'
' "timestamp_ms" : '
'1666790941607,\n'
' "time" : "Wed Oct 26 15:29:01 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1575713792,\n'
' "io_kbytes" : 1538783,\n'
' "bw_bytes" : 225069817,\n'
' "bw" : 219794,\n'
' "iops" : 439588.344522,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3077558,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1442,\n'
' "max" : 20608,\n'
' "mean" : 1603.968562,\n'
' "stddev" : '
'112.982422,\n'
' "N" : 3077559\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4729,\n'
' "max" : 131707,\n'
' "mean" : '
'16357.507495,\n'
' "stddev" : '
'3033.941500,\n'
' "N" : 3077565,\n'
' "percentile" : {\n'
' "1.000000" : 13632,\n'
' "5.000000" : 15552,\n'
' "10.000000" : '
'15552,\n'
' "20.000000" : '
'15680,\n'
' "30.000000" : '
'15680,\n'
' "40.000000" : '
'15808,\n'
' "50.000000" : '
'15808,\n'
' "60.000000" : '
'15808,\n'
' "70.000000" : '
'15936,\n'
' "80.000000" : '
'16320,\n'
' "90.000000" : '
'16512,\n'
' "95.000000" : '
'17536,\n'
' "99.000000" : '
'31872,\n'
' "99.500000" : '
'36096,\n'
' "99.900000" : '
'49408,\n'
' "99.950000" : '
'64256,\n'
' "99.990000" : 77312\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6242,\n'
' "max" : 133490,\n'
' "mean" : '
'17961.475945,\n'
' "stddev" : '
'3035.114182,\n'
' "N" : 3077565\n'
' },\n'
' "bw_min" : 219076,\n'
' "bw_max" : 222226,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'219857.071429,\n'
' "bw_dev" : 825.530735,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 438152,\n'
' "iops_max" : 444452,\n'
' "iops_mean" : '
'439714.285714,\n'
' "iops_stddev" : '
'1651.104219,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 30.700000,\n'
' "sys_cpu" : 67.271429,\n'
' "ctx" : 18039,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.440284,\n'
' "20" : 95.900321,\n'
' "50" : 3.560713,\n'
' "100" : 0.098650,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 4443543,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 25340,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 25340,\n'
' "util" : 99.069123\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=11259: Wed '
'Oct 26 15:29:01 2022\n'
' read: IOPS=440k, BW=215MiB/s '
'(225MB/s)(1503MiB/7001msec)\n'
' slat (nsec): min=1442, '
'max=20608, avg=1603.97, '
'stdev=112.98\n'
' clat (usec): min=4, max=131, '
'avg=16.36, stdev= 3.03\n'
' lat (usec): min=6, max=133, '
'avg=17.96, stdev= 3.04\n'
' clat percentiles (nsec):\n'
' | 1.00th=[13632], '
'5.00th=[15552], 10.00th=[15552], '
'20.00th=[15680],\n'
' | 30.00th=[15680], '
'40.00th=[15808], '
'50.00th=[15808], '
'60.00th=[15808],\n'
' | 70.00th=[15936], '
'80.00th=[16320], '
'90.00th=[16512], '
'95.00th=[17536],\n'
' | 99.00th=[31872], '
'99.50th=[36096], '
'99.90th=[49408], '
'99.95th=[64256],\n'
' | 99.99th=[77312]\n'
' bw ( KiB/s): min=219076, '
'max=222226, per=100.00%, '
'avg=219857.07, stdev=825.53, '
'samples=14\n'
' iops : min=438152, '
'max=444452, avg=439714.29, '
'stdev=1651.10, samples=14\n'
' lat (usec) : 10=0.44%, '
'20=95.90%, 50=3.56%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=30.70%, '
'sys=67.27%, ctx=18039, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3077558,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=215MiB/s (225MB/s), '
'215MiB/s-215MiB/s '
'(225MB/s-225MB/s), io=1503MiB '
'(1576MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=4443543/0, '
'merge=0/0, ticks=25340/0, '
'in_queue=25340, util=99.07%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_369.output'),
'state': {'begin': 1666783851.5465963,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 2.8698582649230957,
'end': 1666783854.4164546,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_369.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_369.state')},
'cmd_37': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_37.output'),
'state': {'begin': 1666782217.7051482,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.85337233543396,
'end': 1666782218.5585206,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_37.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_37.state')},
'cmd_370': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_libaio-bdev_aio_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_370.output'),
'state': {'begin': 1666783857.8533573,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.9677407741546631,
'end': 1666783858.821098,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_370.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_370.state')},
'cmd_371': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_371.output'),
'state': {'begin': 1666783858.8232763,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-bdev_aio_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.803528547286987,
'end': 1666783870.6268048,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_371.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_371.state')},
'cmd_372': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790960,\n'
' "timestamp_ms" : '
'1666790960594,\n'
' "time" : "Wed Oct 26 15:29:20 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1661046784,\n'
' "io_kbytes" : 1622116,\n'
' "bw_bytes" : 237258503,\n'
' "bw" : 231697,\n'
' "iops" : 463394.372233,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3244224,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1633,\n'
' "max" : 22713,\n'
' "mean" : 1818.265835,\n'
' "stddev" : '
'123.789028,\n'
' "N" : 3244225\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 361,\n'
' "max" : 109205,\n'
' "mean" : '
'10578.592328,\n'
' "stddev" : '
'4535.960378,\n'
' "N" : 3244224,\n'
' "percentile" : {\n'
' "1.000000" : 4192,\n'
' "5.000000" : 4320,\n'
' "10.000000" : 6112,\n'
' "20.000000" : 6304,\n'
' "30.000000" : 8160,\n'
' "40.000000" : 8384,\n'
' "50.000000" : '
'10432,\n'
' "60.000000" : '
'11200,\n'
' "70.000000" : '
'12864,\n'
' "80.000000" : '
'14528,\n'
' "90.000000" : '
'14784,\n'
' "95.000000" : '
'15296,\n'
' "99.000000" : '
'26496,\n'
' "99.500000" : '
'28288,\n'
' "99.900000" : '
'43776,\n'
' "99.950000" : '
'60160,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5740,\n'
' "max" : 111099,\n'
' "mean" : '
'12396.858125,\n'
' "stddev" : '
'4531.767359,\n'
' "N" : 3244224\n'
' },\n'
' "bw_min" : 230450,\n'
' "bw_max" : 233465,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'231740.142857,\n'
' "bw_dev" : 820.889841,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 460900,\n'
' "iops_max" : 466930,\n'
' "iops_mean" : '
'463480.285714,\n'
' "iops_stddev" : '
'1641.459736,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 36.971429,\n'
' "sys_cpu" : 63.014286,\n'
' "ctx" : 5,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 8.133347,\n'
' "4" : 72.486425,\n'
' "8" : 19.379550,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 85.202722,\n'
' "8" : 14.797278,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 43.187308,\n'
' "20" : 54.000433,\n'
' "50" : 2.732826,\n'
' "100" : 0.077307,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=11316: Wed '
'Oct 26 15:29:20 2022\n'
' read: IOPS=463k, BW=226MiB/s '
'(237MB/s)(1584MiB/7001msec)\n'
' slat (nsec): min=1633, '
'max=22713, avg=1818.27, '
'stdev=123.79\n'
' clat (nsec): min=361, '
'max=109205, avg=10578.59, '
'stdev=4535.96\n'
' lat (usec): min=5, max=111, '
'avg=12.40, stdev= 4.53\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4192], '
'5.00th=[ 4320], 10.00th=[ 6112], '
'20.00th=[ 6304],\n'
' | 30.00th=[ 8160], '
'40.00th=[ 8384], '
'50.00th=[10432], '
'60.00th=[11200],\n'
' | 70.00th=[12864], '
'80.00th=[14528], '
'90.00th=[14784], '
'95.00th=[15296],\n'
' | 99.00th=[26496], '
'99.50th=[28288], '
'99.90th=[43776], '
'99.95th=[60160],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=230450, '
'max=233465, per=100.00%, '
'avg=231740.14, stdev=820.89, '
'samples=14\n'
' iops : min=460900, '
'max=466930, avg=463480.29, '
'stdev=1641.46, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=43.19%, 20=54.00%, 50=2.73%, '
'100=0.08%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=36.97%, '
'sys=63.01%, ctx=5, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=8.1%, '
'4=72.5%, 8=19.4%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=85.2%, 8=14.8%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3244224,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=226MiB/s (237MB/s), '
'226MiB/s-226MiB/s '
'(237MB/s-237MB/s), io=1584MiB '
'(1661MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_372.output'),
'state': {'begin': 1666783870.6286519,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 1.4652798175811768,
'end': 1666783872.0939317,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_372.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_372.state')},
'cmd_373': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_libaio-bdev_aio_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_373.output'),
'state': {'begin': 1666783874.6259732,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 1.0284993648529053,
'end': 1666783875.6544726,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_373.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_373.state')},
'cmd_374': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_374.output'),
'state': {'begin': 1666783875.6564553,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-bdev_aio_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.919304609298706,
'end': 1666783887.57576,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_374.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_374.state')},
'cmd_375': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790977,\n'
' "timestamp_ms" : '
'1666790977538,\n'
' "time" : "Wed Oct 26 15:29:37 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1666486272,\n'
' "io_kbytes" : 1627428,\n'
' "bw_bytes" : 238035462,\n'
' "bw" : 232456,\n'
' "iops" : 464912.012570,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3254849,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1623,\n'
' "max" : 23605,\n'
' "mean" : 1803.010349,\n'
' "stddev" : '
'126.458209,\n'
' "N" : 3254850\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 370,\n'
' "max" : 125115,\n'
' "mean" : '
'10935.840299,\n'
' "stddev" : '
'4372.129142,\n'
' "N" : 3254856,\n'
' "percentile" : {\n'
' "1.000000" : 4256,\n'
' "5.000000" : 6112,\n'
' "10.000000" : 6176,\n'
' "20.000000" : 6560,\n'
' "30.000000" : 8160,\n'
' "40.000000" : '
'10048,\n'
' "50.000000" : '
'10688,\n'
' "60.000000" : '
'12480,\n'
' "70.000000" : '
'12736,\n'
' "80.000000" : '
'14400,\n'
' "90.000000" : '
'14656,\n'
' "95.000000" : '
'15168,\n'
' "99.000000" : '
'27008,\n'
' "99.500000" : '
'28800,\n'
' "99.900000" : '
'43264,\n'
' "99.950000" : '
'60672,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5661,\n'
' "max" : 126928,\n'
' "mean" : '
'12738.850612,\n'
' "stddev" : '
'4363.905228,\n'
' "N" : 3254856\n'
' },\n'
' "bw_min" : 229362,\n'
' "bw_max" : 234011,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'232498.785714,\n'
' "bw_dev" : 1354.748752,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 458724,\n'
' "iops_max" : 468022,\n'
' "iops_mean" : '
'464997.642857,\n'
' "iops_stddev" : '
'2709.526147,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 41.585714,\n'
' "sys_cpu" : 58.385714,\n'
' "ctx" : 7,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 1.864664,\n'
' "4" : 77.562246,\n'
' "8" : 20.571922,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 84.411387,\n'
' "8" : 15.588613,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 39.686818,\n'
' "20" : 57.371356,\n'
' "50" : 2.863297,\n'
' "100" : 0.075702,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=11373: Wed '
'Oct 26 15:29:37 2022\n'
' read: IOPS=465k, BW=227MiB/s '
'(238MB/s)(1589MiB/7001msec)\n'
' slat (nsec): min=1623, '
'max=23605, avg=1803.01, '
'stdev=126.46\n'
' clat (nsec): min=370, '
'max=125115, avg=10935.84, '
'stdev=4372.13\n'
' lat (usec): min=5, max=126, '
'avg=12.74, stdev= 4.36\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4256], '
'5.00th=[ 6112], 10.00th=[ 6176], '
'20.00th=[ 6560],\n'
' | 30.00th=[ 8160], '
'40.00th=[10048], '
'50.00th=[10688], '
'60.00th=[12480],\n'
' | 70.00th=[12736], '
'80.00th=[14400], '
'90.00th=[14656], '
'95.00th=[15168],\n'
' | 99.00th=[27008], '
'99.50th=[28800], '
'99.90th=[43264], '
'99.95th=[60672],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=229362, '
'max=234011, per=100.00%, '
'avg=232498.79, stdev=1354.75, '
'samples=14\n'
' iops : min=458724, '
'max=468022, avg=464997.64, '
'stdev=2709.53, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=39.69%, 20=57.37%, 50=2.86%, '
'100=0.08%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=41.59%, '
'sys=58.39%, ctx=7, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=1.9%, '
'4=77.6%, 8=20.6%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=84.4%, 8=15.6%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3254849,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=227MiB/s (238MB/s), '
'227MiB/s-227MiB/s '
'(238MB/s-238MB/s), io=1589MiB '
'(1666MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_375.output'),
'state': {'begin': 1666783887.5784094,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.8194758892059326,
'end': 1666783888.3978853,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_375.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_375.state')},
'cmd_376': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_libaio-bdev_aio_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_376.output'),
'state': {'begin': 1666783890.4773853,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.8124725818634033,
'end': 1666783891.2898579,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_376.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_376.state')},
'cmd_377': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_377.output'),
'state': {'begin': 1666783891.2919939,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-bdev_aio_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.5554780960083,
'end': 1666783902.847472,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_377.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_377.state')},
'cmd_378': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666790992,\n'
' "timestamp_ms" : '
'1666790992810,\n'
' "time" : "Wed Oct 26 15:29:52 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1667806208,\n'
' "io_kbytes" : 1628717,\n'
' "bw_bytes" : 238223997,\n'
' "bw" : 232640,\n'
' "iops" : 465280.531353,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3257429,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1623,\n'
' "max" : 23795,\n'
' "mean" : 1807.492342,\n'
' "stddev" : '
'121.710490,\n'
' "N" : 3257430\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 391,\n'
' "max" : 110808,\n'
' "mean" : '
'10528.263130,\n'
' "stddev" : '
'4533.572349,\n'
' "N" : 3257434,\n'
' "percentile" : {\n'
' "1.000000" : 4192,\n'
' "5.000000" : 4256,\n'
' "10.000000" : 6112,\n'
' "20.000000" : 6240,\n'
' "30.000000" : 8096,\n'
' "40.000000" : 8256,\n'
' "50.000000" : '
'10304,\n'
' "60.000000" : '
'11200,\n'
' "70.000000" : '
'12736,\n'
' "80.000000" : '
'14528,\n'
' "90.000000" : '
'14784,\n'
' "95.000000" : '
'15296,\n'
' "99.000000" : '
'26240,\n'
' "99.500000" : '
'28288,\n'
' "99.900000" : '
'43264,\n'
' "99.950000" : '
'60672,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5650,\n'
' "max" : 112531,\n'
' "mean" : '
'12335.755434,\n'
' "stddev" : '
'4529.016440,\n'
' "N" : 3257434\n'
' },\n'
' "bw_min" : 230862,\n'
' "bw_max" : 235114,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'232693.000000,\n'
' "bw_dev" : 1547.197367,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 461716,\n'
' "iops_max" : 470228,\n'
' "iops_mean" : '
'465386.000000,\n'
' "iops_stddev" : '
'3095.091101,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 38.057143,\n'
' "sys_cpu" : 61.928571,\n'
' "ctx" : 4,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 8.325922,\n'
' "4" : 72.295359,\n'
' "8" : 19.377736,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 85.258211,\n'
' "8" : 14.741789,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 43.861094,\n'
' "20" : 53.338047,\n'
' "50" : 2.724081,\n'
' "100" : 0.074261,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=11431: Wed '
'Oct 26 15:29:52 2022\n'
' read: IOPS=465k, BW=227MiB/s '
'(238MB/s)(1591MiB/7001msec)\n'
' slat (nsec): min=1623, '
'max=23795, avg=1807.49, '
'stdev=121.71\n'
' clat (nsec): min=391, '
'max=110808, avg=10528.26, '
'stdev=4533.57\n'
' lat (usec): min=5, max=112, '
'avg=12.34, stdev= 4.53\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4192], '
'5.00th=[ 4256], 10.00th=[ 6112], '
'20.00th=[ 6240],\n'
' | 30.00th=[ 8096], '
'40.00th=[ 8256], '
'50.00th=[10304], '
'60.00th=[11200],\n'
' | 70.00th=[12736], '
'80.00th=[14528], '
'90.00th=[14784], '
'95.00th=[15296],\n'
' | 99.00th=[26240], '
'99.50th=[28288], '
'99.90th=[43264], '
'99.95th=[60672],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=230862, '
'max=235114, per=100.00%, '
'avg=232693.00, stdev=1547.20, '
'samples=14\n'
' iops : min=461716, '
'max=470228, avg=465386.00, '
'stdev=3095.09, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=43.86%, 20=53.34%, 50=2.72%, '
'100=0.07%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=38.06%, '
'sys=61.93%, ctx=4, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=8.3%, '
'4=72.3%, 8=19.4%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=85.3%, 8=14.7%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3257429,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=227MiB/s (238MB/s), '
'227MiB/s-227MiB/s '
'(238MB/s-238MB/s), io=1591MiB '
'(1668MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_378.output'),
'state': {'begin': 1666783902.8541772,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.8841710090637207,
'end': 1666783903.7383482,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_378.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_378.state')},
'cmd_379': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_379.output'),
'state': {'begin': 1666783905.848171,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8557703495025635,
'end': 1666783906.7039413,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_379.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_379.state')},
'cmd_38': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_38.output'),
'state': {'begin': 1666782218.5652142,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.326873302459717,
'end': 1666782229.8920875,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_38.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_38.state')},
'cmd_380': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_380.output'),
'state': {'begin': 1666783906.7059243,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.521454334259033,
'end': 1666783918.2273786,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_380.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_380.state')},
'cmd_381': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666791008,\n'
' "timestamp_ms" : '
'1666791008196,\n'
' "time" : "Wed Oct 26 15:30:08 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1567925248,\n'
' "io_kbytes" : 1531177,\n'
' "bw_bytes" : 223957327,\n'
' "bw" : 218708,\n'
' "iops" : 437415.940580,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3062349,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1663,\n'
' "max" : 34314,\n'
' "mean" : 1858.695535,\n'
' "stddev" : '
'122.078812,\n'
' "N" : 3062350\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 831,\n'
' "max" : 122730,\n'
' "mean" : '
'11448.851285,\n'
' "stddev" : '
'4591.776289,\n'
' "N" : 3062354,\n'
' "percentile" : {\n'
' "1.000000" : 4704,\n'
' "5.000000" : 4896,\n'
' "10.000000" : 6688,\n'
' "20.000000" : 6880,\n'
' "30.000000" : 8768,\n'
' "40.000000" : 9152,\n'
' "50.000000" : '
'11584,\n'
' "60.000000" : '
'12352,\n'
' "70.000000" : '
'13888,\n'
' "80.000000" : '
'15680,\n'
' "90.000000" : '
'16064,\n'
' "95.000000" : '
'16768,\n'
' "99.000000" : '
'26240,\n'
' "99.500000" : '
'29056,\n'
' "99.900000" : '
'44800,\n'
' "99.950000" : '
'59648,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5661,\n'
' "max" : 124534,\n'
' "mean" : '
'13307.546791,\n'
' "stddev" : '
'4584.897019,\n'
' "N" : 3062354\n'
' },\n'
' "bw_min" : 218159,\n'
' "bw_max" : 219551,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'218814.142857,\n'
' "bw_dev" : 384.929535,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 436318,\n'
' "iops_max" : 439102,\n'
' "iops_mean" : '
'437628.428571,\n'
' "iops_stddev" : '
'769.682623,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 39.185714,\n'
' "sys_cpu" : 60.785714,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 7.306744,\n'
' "4" : 73.195772,\n'
' "8" : 19.496700,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 85.097695,\n'
' "8" : 14.902305,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 42.621955,\n'
' "20" : 54.869220,\n'
' "50" : 2.423107,\n'
' "100" : 0.082584,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=11488: Wed '
'Oct 26 15:30:08 2022\n'
' read: IOPS=437k, BW=214MiB/s '
'(224MB/s)(1495MiB/7001msec)\n'
' slat (nsec): min=1663, '
'max=34314, avg=1858.70, '
'stdev=122.08\n'
' clat (nsec): min=831, '
'max=122730, avg=11448.85, '
'stdev=4591.78\n'
' lat (usec): min=5, max=124, '
'avg=13.31, stdev= 4.58\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4704], '
'5.00th=[ 4896], 10.00th=[ 6688], '
'20.00th=[ 6880],\n'
' | 30.00th=[ 8768], '
'40.00th=[ 9152], '
'50.00th=[11584], '
'60.00th=[12352],\n'
' | 70.00th=[13888], '
'80.00th=[15680], '
'90.00th=[16064], '
'95.00th=[16768],\n'
' | 99.00th=[26240], '
'99.50th=[29056], '
'99.90th=[44800], '
'99.95th=[59648],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=218159, '
'max=219551, per=100.00%, '
'avg=218814.14, stdev=384.93, '
'samples=14\n'
' iops : min=436318, '
'max=439102, avg=437628.43, '
'stdev=769.68, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=42.62%, 20=54.87%, 50=2.42%, '
'100=0.08%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=39.19%, '
'sys=60.79%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=7.3%, '
'4=73.2%, 8=19.5%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=85.1%, 8=14.9%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3062349,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=214MiB/s (224MB/s), '
'214MiB/s-214MiB/s '
'(224MB/s-224MB/s), io=1495MiB '
'(1568MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_381.output'),
'state': {'begin': 1666783918.2299788,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.828787088394165,
'end': 1666783919.058766,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_381.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_381.state')},
'cmd_382': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_382.output'),
'state': {'begin': 1666783921.2030697,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8114423751831055,
'end': 1666783922.014512,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_382.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_382.state')},
'cmd_383': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_383.output'),
'state': {'begin': 1666783922.0164816,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.534029483795166,
'end': 1666783933.5505111,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_383.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_383.state')},
'cmd_384': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666791023,\n'
' "timestamp_ms" : '
'1666791023513,\n'
' "time" : "Wed Oct 26 15:30:23 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1576878592,\n'
' "io_kbytes" : 1539920,\n'
' "bw_bytes" : 225236193,\n'
' "bw" : 219957,\n'
' "iops" : 439913.726610,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3079836,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1663,\n'
' "max" : 19526,\n'
' "mean" : 1853.081294,\n'
' "stddev" : '
'134.461061,\n'
' "N" : 3079837\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 812,\n'
' "max" : 111990,\n'
' "mean" : '
'11416.888518,\n'
' "stddev" : '
'4588.974480,\n'
' "N" : 3079841,\n'
' "percentile" : {\n'
' "1.000000" : 4704,\n'
' "5.000000" : 4896,\n'
' "10.000000" : 6624,\n'
' "20.000000" : 6816,\n'
' "30.000000" : 8768,\n'
' "40.000000" : 9152,\n'
' "50.000000" : '
'11584,\n'
' "60.000000" : '
'12352,\n'
' "70.000000" : '
'13888,\n'
' "80.000000" : '
'15552,\n'
' "90.000000" : '
'15936,\n'
' "95.000000" : '
'16512,\n'
' "99.000000" : '
'26752,\n'
' "99.500000" : '
'29312,\n'
' "99.900000" : '
'44288,\n'
' "99.950000" : '
'60160,\n'
' "99.990000" : 73216\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5660,\n'
' "max" : 113774,\n'
' "mean" : '
'13269.969731,\n'
' "stddev" : '
'4580.597825,\n'
' "N" : 3079841\n'
' },\n'
' "bw_min" : 219083,\n'
' "bw_max" : 221465,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'219996.928571,\n'
' "bw_dev" : 649.316737,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 438166,\n'
' "iops_max" : 442931,\n'
' "iops_mean" : '
'439993.928571,\n'
' "iops_stddev" : '
'1298.760028,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 37.185714,\n'
' "sys_cpu" : 62.800000,\n'
' "ctx" : 5,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 6.747048,\n'
' "4" : 73.657980,\n'
' "8" : 19.593803,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 85.015234,\n'
' "8" : 14.984766,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 42.298421,\n'
' "20" : 55.114980,\n'
' "50" : 2.504776,\n'
' "100" : 0.078511,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=11545: Wed '
'Oct 26 15:30:23 2022\n'
' read: IOPS=440k, BW=215MiB/s '
'(225MB/s)(1504MiB/7001msec)\n'
' slat (nsec): min=1663, '
'max=19526, avg=1853.08, '
'stdev=134.46\n'
' clat (nsec): min=812, '
'max=111990, avg=11416.89, '
'stdev=4588.97\n'
' lat (usec): min=5, max=113, '
'avg=13.27, stdev= 4.58\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4704], '
'5.00th=[ 4896], 10.00th=[ 6624], '
'20.00th=[ 6816],\n'
' | 30.00th=[ 8768], '
'40.00th=[ 9152], '
'50.00th=[11584], '
'60.00th=[12352],\n'
' | 70.00th=[13888], '
'80.00th=[15552], '
'90.00th=[15936], '
'95.00th=[16512],\n'
' | 99.00th=[26752], '
'99.50th=[29312], '
'99.90th=[44288], '
'99.95th=[60160],\n'
' | 99.99th=[73216]\n'
' bw ( KiB/s): min=219083, '
'max=221465, per=100.00%, '
'avg=219996.93, stdev=649.32, '
'samples=14\n'
' iops : min=438166, '
'max=442931, avg=439993.93, '
'stdev=1298.76, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=42.30%, 20=55.11%, 50=2.50%, '
'100=0.08%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=37.19%, '
'sys=62.80%, ctx=5, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=6.7%, '
'4=73.7%, 8=19.6%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=85.0%, 8=15.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3079836,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=215MiB/s (225MB/s), '
'215MiB/s-215MiB/s '
'(225MB/s-225MB/s), io=1504MiB '
'(1577MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_384.output'),
'state': {'begin': 1666783933.554262,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8546414375305176,
'end': 1666783934.4089034,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_384.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_384.state')},
'cmd_385': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_385.output'),
'state': {'begin': 1666783936.581162,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8732352256774902,
'end': 1666783937.4543972,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_385.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_385.state')},
'cmd_386': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_386.output'),
'state': {'begin': 1666783937.4561808,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.624598264694214,
'end': 1666783949.080779,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_386.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_386.state')},
'cmd_387': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666791039,\n'
' "timestamp_ms" : '
'1666791039010,\n'
' "time" : "Wed Oct 26 15:30:39 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1597813760,\n'
' "io_kbytes" : 1560365,\n'
' "bw_bytes" : 228226504,\n'
' "bw" : 222877,\n'
' "iops" : 445754.320811,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3120726,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1653,\n'
' "max" : 42329,\n'
' "mean" : 1834.530434,\n'
' "stddev" : '
'111.510378,\n'
' "N" : 3120727\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 791,\n'
' "max" : 125785,\n'
' "mean" : '
'11024.351078,\n'
' "stddev" : '
'4765.287623,\n'
' "N" : 3120730,\n'
' "percentile" : {\n'
' "1.000000" : 4640,\n'
' "5.000000" : 4704,\n'
' "10.000000" : 4896,\n'
' "20.000000" : 6688,\n'
' "30.000000" : 8640,\n'
' "40.000000" : 8768,\n'
' "50.000000" : '
'10688,\n'
' "60.000000" : '
'11712,\n'
' "70.000000" : '
'13632,\n'
' "80.000000" : '
'15424,\n'
' "90.000000" : '
'15808,\n'
' "95.000000" : '
'16320,\n'
' "99.000000" : '
'27520,\n'
' "99.500000" : '
'29824,\n'
' "99.900000" : '
'45312,\n'
' "99.950000" : '
'61184,\n'
' "99.990000" : 74240\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5721,\n'
' "max" : 127559,\n'
' "mean" : '
'12858.881482,\n'
' "stddev" : '
'4762.175625,\n'
' "N" : 3120730\n'
' },\n'
' "bw_min" : 221033,\n'
' "bw_max" : 224499,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'222992.500000,\n'
' "bw_dev" : 1021.068049,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 442066,\n'
' "iops_max" : 448998,\n'
' "iops_mean" : '
'445985.000000,\n'
' "iops_stddev" : '
'2042.136099,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 36.557143,\n'
' "sys_cpu" : 63.414286,\n'
' "ctx" : 13,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 10.780793,\n'
' "4" : 70.423421,\n'
' "8" : 18.795178,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 85.618620,\n'
' "8" : 14.381380,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 44.750004,\n'
' "20" : 52.543575,\n'
' "50" : 2.621249,\n'
' "100" : 0.082577,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=11603: Wed '
'Oct 26 15:30:39 2022\n'
' read: IOPS=446k, BW=218MiB/s '
'(228MB/s)(1524MiB/7001msec)\n'
' slat (nsec): min=1653, '
'max=42329, avg=1834.53, '
'stdev=111.51\n'
' clat (nsec): min=791, '
'max=125785, avg=11024.35, '
'stdev=4765.29\n'
' lat (usec): min=5, max=127, '
'avg=12.86, stdev= 4.76\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4640], '
'5.00th=[ 4704], 10.00th=[ 4896], '
'20.00th=[ 6688],\n'
' | 30.00th=[ 8640], '
'40.00th=[ 8768], '
'50.00th=[10688], '
'60.00th=[11712],\n'
' | 70.00th=[13632], '
'80.00th=[15424], '
'90.00th=[15808], '
'95.00th=[16320],\n'
' | 99.00th=[27520], '
'99.50th=[29824], '
'99.90th=[45312], '
'99.95th=[61184],\n'
' | 99.99th=[74240]\n'
' bw ( KiB/s): min=221033, '
'max=224499, per=100.00%, '
'avg=222992.50, stdev=1021.07, '
'samples=14\n'
' iops : min=442066, '
'max=448998, avg=445985.00, '
'stdev=2042.14, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=44.75%, 20=52.54%, 50=2.62%, '
'100=0.08%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=36.56%, '
'sys=63.41%, ctx=13, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=10.8%, 4=70.4%, 8=18.8%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=85.6%, 8=14.4%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3120726,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=218MiB/s (228MB/s), '
'218MiB/s-218MiB/s '
'(228MB/s-228MB/s), io=1524MiB '
'(1598MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_387.output'),
'state': {'begin': 1666783949.083425,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8751075267791748,
'end': 1666783949.9585326,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_387.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_387.state')},
'cmd_388': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_libaio-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_388.output'),
'state': {'begin': 1666783951.007023,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8368914127349854,
'end': 1666783951.8439145,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_388.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_388.state')},
'cmd_389': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_389.output'),
'state': {'begin': 1666783951.845953,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.118852138519287,
'end': 1666783962.9648051,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_389.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_389.state')},
'cmd_39': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789319,\n'
' "timestamp_ms" : '
'1666789319922,\n'
' "time" : "Wed Oct 26 15:01:59 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 484563456,\n'
' "io_kbytes" : 473206,\n'
' "bw_bytes" : 69213463,\n'
' "bw" : 67591,\n'
' "iops" : 135182.545351,\n'
' "runtime" : 7001,\n'
' "total_ios" : 946413,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 150,\n'
' "max" : 9447,\n'
' "mean" : 172.883611,\n'
' "stddev" : 23.288240,\n'
' "N" : 946413\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 6261,\n'
' "max" : 85470,\n'
' "mean" : 6992.056989,\n'
' "stddev" : '
'1186.274005,\n'
' "N" : 946413,\n'
' "percentile" : {\n'
' "1.000000" : 6496,\n'
' "5.000000" : 6688,\n'
' "10.000000" : 6752,\n'
' "20.000000" : 6752,\n'
' "30.000000" : 6752,\n'
' "40.000000" : 6816,\n'
' "50.000000" : 6816,\n'
' "60.000000" : 6880,\n'
' "70.000000" : 6880,\n'
' "80.000000" : 6944,\n'
' "90.000000" : 7008,\n'
' "95.000000" : 7136,\n'
' "99.000000" : 13120,\n'
' "99.500000" : 14784,\n'
' "99.900000" : 17024,\n'
' "99.950000" : 17280,\n'
' "99.990000" : 60672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6432,\n'
' "max" : 85640,\n'
' "mean" : 7164.940600,\n'
' "stddev" : '
'1186.749310,\n'
' "N" : 946413\n'
' },\n'
' "bw_min" : 67403,\n'
' "bw_max" : 67839,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'67627.785714,\n'
' "bw_dev" : 111.226850,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 134806,\n'
' "iops_max" : 135679,\n'
' "iops_mean" : '
'135255.642857,\n'
' "iops_stddev" : '
'222.599885,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 53.014286,\n'
' "sys_cpu" : 46.957143,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 98.381151,\n'
' "20" : 1.599090,\n'
' "50" : 0.010000,\n'
' "100" : 0.015955,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=5540: Wed '
'Oct 26 15:01:59 2022\n'
' read: IOPS=135k, BW=66.0MiB/s '
'(69.2MB/s)(462MiB/7001msec)\n'
' slat (nsec): min=150, '
'max=9447, avg=172.88, '
'stdev=23.29\n'
' clat (nsec): min=6261, '
'max=85470, avg=6992.06, '
'stdev=1186.27\n'
' lat (nsec): min=6432, '
'max=85640, avg=7164.94, '
'stdev=1186.75\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6496], 5.00th=[ '
'6688], 10.00th=[ 6752], 20.00th=[ '
'6752],\n'
' | 30.00th=[ 6752], 40.00th=[ '
'6816], 50.00th=[ 6816], 60.00th=[ '
'6880],\n'
' | 70.00th=[ 6880], 80.00th=[ '
'6944], 90.00th=[ 7008], 95.00th=[ '
'7136],\n'
' | 99.00th=[13120], '
'99.50th=[14784], 99.90th=[17024], '
'99.95th=[17280],\n'
' | 99.99th=[60672]\n'
' bw ( KiB/s): min=67403, '
'max=67839, per=100.00%, '
'avg=67627.79, stdev=111.23, '
'samples=14\n'
' iops : min=134806, '
'max=135679, avg=135255.64, '
'stdev=222.60, samples=14\n'
' lat (usec) : 10=98.38%, '
'20=1.60%, 50=0.01%, 100=0.02%\n'
' cpu : usr=53.01%, '
'sys=46.96%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=946413,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=66.0MiB/s (69.2MB/s), '
'66.0MiB/s-66.0MiB/s '
'(69.2MB/s-69.2MB/s), io=462MiB '
'(485MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_39.output'),
'state': {'begin': 1666782229.8945956,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.827765703201294,
'end': 1666782230.7223613,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_39.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_39.state')},
'cmd_390': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666791053,\n'
' "timestamp_ms" : '
'1666791053029,\n'
' "time" : "Wed Oct 26 15:30:53 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1495874560,\n'
' "io_kbytes" : 1460815,\n'
' "bw_bytes" : 213665842,\n'
' "bw" : 208658,\n'
' "iops" : 417315.097843,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2921623,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1512,\n'
' "max" : 36950,\n'
' "mean" : 1657.647678,\n'
' "stddev" : '
'106.702955,\n'
' "N" : 2921623\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5039,\n'
' "max" : 134552,\n'
' "mean" : '
'17269.205602,\n'
' "stddev" : '
'2767.694361,\n'
' "N" : 2921630,\n'
' "percentile" : {\n'
' "1.000000" : 14912,\n'
' "5.000000" : 16512,\n'
' "10.000000" : '
'16512,\n'
' "20.000000" : '
'16768,\n'
' "30.000000" : '
'16768,\n'
' "40.000000" : '
'16768,\n'
' "50.000000" : '
'16768,\n'
' "60.000000" : '
'16768,\n'
' "70.000000" : '
'16768,\n'
' "80.000000" : '
'17280,\n'
' "90.000000" : '
'17536,\n'
' "95.000000" : '
'18048,\n'
' "99.000000" : '
'30336,\n'
' "99.500000" : '
'34560,\n'
' "99.900000" : '
'50944,\n'
' "99.950000" : '
'67072,\n'
' "99.990000" : 80384\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6592,\n'
' "max" : 136265,\n'
' "mean" : '
'18926.853252,\n'
' "stddev" : '
'2770.487130,\n'
' "N" : 2921630\n'
' },\n'
' "bw_min" : 208125,\n'
' "bw_max" : 209317,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'208769.428571,\n'
' "bw_dev" : 418.942389,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 416250,\n'
' "iops_max" : 418634,\n'
' "iops_mean" : '
'417538.857143,\n'
' "iops_stddev" : '
'837.884778,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 34.928571,\n'
' "sys_cpu" : 63.642857,\n'
' "ctx" : 16304,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999983,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.213888,\n'
' "20" : 96.208785,\n'
' "50" : 3.475089,\n'
' "100" : 0.101450,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=11651: Wed '
'Oct 26 15:30:53 2022\n'
' read: IOPS=417k, BW=204MiB/s '
'(214MB/s)(1427MiB/7001msec)\n'
' slat (nsec): min=1512, '
'max=36950, avg=1657.65, '
'stdev=106.70\n'
' clat (usec): min=5, max=134, '
'avg=17.27, stdev= 2.77\n'
' lat (usec): min=6, max=136, '
'avg=18.93, stdev= 2.77\n'
' clat percentiles (nsec):\n'
' | 1.00th=[14912], '
'5.00th=[16512], 10.00th=[16512], '
'20.00th=[16768],\n'
' | 30.00th=[16768], '
'40.00th=[16768], '
'50.00th=[16768], '
'60.00th=[16768],\n'
' | 70.00th=[16768], '
'80.00th=[17280], '
'90.00th=[17536], '
'95.00th=[18048],\n'
' | 99.00th=[30336], '
'99.50th=[34560], '
'99.90th=[50944], '
'99.95th=[67072],\n'
' | 99.99th=[80384]\n'
' bw ( KiB/s): min=208125, '
'max=209317, per=100.00%, '
'avg=208769.43, stdev=418.94, '
'samples=14\n'
' iops : min=416250, '
'max=418634, avg=417538.86, '
'stdev=837.88, samples=14\n'
' lat (usec) : 10=0.21%, '
'20=96.21%, 50=3.48%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=34.93%, '
'sys=63.64%, ctx=16304, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2921623,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=204MiB/s (214MB/s), '
'204MiB/s-204MiB/s '
'(214MB/s-214MB/s), io=1427MiB '
'(1496MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_390.output'),
'state': {'begin': 1666783962.9672246,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8798742294311523,
'end': 1666783963.8470988,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_390.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_390.state')},
'cmd_391': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_libaio-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_391.output'),
'state': {'begin': 1666783964.928627,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8847477436065674,
'end': 1666783965.8133748,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_391.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_391.state')},
'cmd_392': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_392.output'),
'state': {'begin': 1666783965.8152633,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.087571859359741,
'end': 1666783976.9028351,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_392.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_392.state')},
'cmd_393': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666791066,\n'
' "timestamp_ms" : '
'1666791066967,\n'
' "time" : "Wed Oct 26 15:31:06 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1492745216,\n'
' "io_kbytes" : 1457759,\n'
' "bw_bytes" : 213218856,\n'
' "bw" : 208221,\n'
' "iops" : 416442.079703,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2915511,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1502,\n'
' "max" : 33263,\n'
' "mean" : 1671.443877,\n'
' "stddev" : '
'126.111968,\n'
' "N" : 2915511\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5090,\n'
' "max" : 134011,\n'
' "mean" : '
'17299.230179,\n'
' "stddev" : '
'2704.916413,\n'
' "N" : 2915518,\n'
' "percentile" : {\n'
' "1.000000" : 15040,\n'
' "5.000000" : 16512,\n'
' "10.000000" : '
'16512,\n'
' "20.000000" : '
'16768,\n'
' "30.000000" : '
'16768,\n'
' "40.000000" : '
'16768,\n'
' "50.000000" : '
'16768,\n'
' "60.000000" : '
'16768,\n'
' "70.000000" : '
'17024,\n'
' "80.000000" : '
'17280,\n'
' "90.000000" : '
'17536,\n'
' "95.000000" : '
'18048,\n'
' "99.000000" : '
'30336,\n'
' "99.500000" : '
'34560,\n'
' "99.900000" : '
'47360,\n'
' "99.950000" : '
'64256,\n'
' "99.990000" : 79360\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6653,\n'
' "max" : 135624,\n'
' "mean" : '
'18970.673988,\n'
' "stddev" : '
'2707.653440,\n'
' "N" : 2915518\n'
' },\n'
' "bw_min" : 207578,\n'
' "bw_max" : 209884,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'208259.785714,\n'
' "bw_dev" : 724.737107,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 415156,\n'
' "iops_max" : 419769,\n'
' "iops_mean" : '
'416519.500000,\n'
' "iops_stddev" : '
'1449.767660,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 30.614286,\n'
' "sys_cpu" : 67.985714,\n'
' "ctx" : 16266,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999983,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.206173,\n'
' "20" : 96.180395,\n'
' "50" : 3.523842,\n'
' "100" : 0.088767,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=11700: Wed '
'Oct 26 15:31:06 2022\n'
' read: IOPS=416k, BW=203MiB/s '
'(213MB/s)(1424MiB/7001msec)\n'
' slat (nsec): min=1502, '
'max=33263, avg=1671.44, '
'stdev=126.11\n'
' clat (usec): min=5, max=134, '
'avg=17.30, stdev= 2.70\n'
' lat (usec): min=6, max=135, '
'avg=18.97, stdev= 2.71\n'
' clat percentiles (nsec):\n'
' | 1.00th=[15040], '
'5.00th=[16512], 10.00th=[16512], '
'20.00th=[16768],\n'
' | 30.00th=[16768], '
'40.00th=[16768], '
'50.00th=[16768], '
'60.00th=[16768],\n'
' | 70.00th=[17024], '
'80.00th=[17280], '
'90.00th=[17536], '
'95.00th=[18048],\n'
' | 99.00th=[30336], '
'99.50th=[34560], '
'99.90th=[47360], '
'99.95th=[64256],\n'
' | 99.99th=[79360]\n'
' bw ( KiB/s): min=207578, '
'max=209884, per=100.00%, '
'avg=208259.79, stdev=724.74, '
'samples=14\n'
' iops : min=415156, '
'max=419769, avg=416519.50, '
'stdev=1449.77, samples=14\n'
' lat (usec) : 10=0.21%, '
'20=96.18%, 50=3.52%, 100=0.09%, '
'250=0.01%\n'
' cpu : usr=30.61%, '
'sys=67.99%, ctx=16266, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2915511,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=203MiB/s (213MB/s), '
'203MiB/s-203MiB/s '
'(213MB/s-213MB/s), io=1424MiB '
'(1493MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_393.output'),
'state': {'begin': 1666783976.9056933,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.856008768081665,
'end': 1666783977.761702,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_393.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_393.state')},
'cmd_394': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_8_libaio-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_394.output'),
'state': {'begin': 1666783978.8074331,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8541994094848633,
'end': 1666783979.6616325,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_394.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_394.state')},
'cmd_395': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_395.output'),
'state': {'begin': 1666783979.6635144,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.125166177749634,
'end': 1666783990.7886806,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_395.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_395.state')},
'cmd_396': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666791080,\n'
' "timestamp_ms" : '
'1666791080852,\n'
' "time" : "Wed Oct 26 15:31:20 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1497201664,\n'
' "io_kbytes" : 1462111,\n'
' "bw_bytes" : 213855401,\n'
' "bw" : 208843,\n'
' "iops" : 417685.187830,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2924214,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1513,\n'
' "max" : 25247,\n'
' "mean" : 1654.304958,\n'
' "stddev" : '
'100.956893,\n'
' "N" : 2924215\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5080,\n'
' "max" : 137948,\n'
' "mean" : '
'17253.342018,\n'
' "stddev" : '
'2675.005836,\n'
' "N" : 2924221,\n'
' "percentile" : {\n'
' "1.000000" : 14784,\n'
' "5.000000" : 16768,\n'
' "10.000000" : '
'16768,\n'
' "20.000000" : '
'16768,\n'
' "30.000000" : '
'16768,\n'
' "40.000000" : '
'16768,\n'
' "50.000000" : '
'16768,\n'
' "60.000000" : '
'16768,\n'
' "70.000000" : '
'17024,\n'
' "80.000000" : '
'17024,\n'
' "90.000000" : '
'17536,\n'
' "95.000000" : '
'17792,\n'
' "99.000000" : '
'29568,\n'
' "99.500000" : '
'33536,\n'
' "99.900000" : '
'50944,\n'
' "99.950000" : '
'66048,\n'
' "99.990000" : 79360\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6703,\n'
' "max" : 139541,\n'
' "mean" : '
'18907.646970,\n'
' "stddev" : '
'2674.174544,\n'
' "N" : 2924221\n'
' },\n'
' "bw_min" : 208370,\n'
' "bw_max" : 209183,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'208880.357143,\n'
' "bw_dev" : 273.334849,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 416741,\n'
' "iops_max" : 418366,\n'
' "iops_mean" : '
'417760.785714,\n'
' "iops_stddev" : '
'546.526117,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 32.342857,\n'
' "sys_cpu" : 66.300000,\n'
' "ctx" : 15925,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999983,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.234285,\n'
' "20" : 96.261628,\n'
' "50" : 3.402384,\n'
' "100" : 0.100574,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=11749: Wed '
'Oct 26 15:31:20 2022\n'
' read: IOPS=418k, BW=204MiB/s '
'(214MB/s)(1428MiB/7001msec)\n'
' slat (nsec): min=1513, '
'max=25247, avg=1654.30, '
'stdev=100.96\n'
' clat (usec): min=5, max=137, '
'avg=17.25, stdev= 2.68\n'
' lat (usec): min=6, max=139, '
'avg=18.91, stdev= 2.67\n'
' clat percentiles (nsec):\n'
' | 1.00th=[14784], '
'5.00th=[16768], 10.00th=[16768], '
'20.00th=[16768],\n'
' | 30.00th=[16768], '
'40.00th=[16768], '
'50.00th=[16768], '
'60.00th=[16768],\n'
' | 70.00th=[17024], '
'80.00th=[17024], '
'90.00th=[17536], '
'95.00th=[17792],\n'
' | 99.00th=[29568], '
'99.50th=[33536], '
'99.90th=[50944], '
'99.95th=[66048],\n'
' | 99.99th=[79360]\n'
' bw ( KiB/s): min=208370, '
'max=209183, per=100.00%, '
'avg=208880.36, stdev=273.33, '
'samples=14\n'
' iops : min=416741, '
'max=418366, avg=417760.79, '
'stdev=546.53, samples=14\n'
' lat (usec) : 10=0.23%, '
'20=96.26%, 50=3.40%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=32.34%, '
'sys=66.30%, ctx=15925, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2924214,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=204MiB/s (214MB/s), '
'204MiB/s-204MiB/s '
'(214MB/s-214MB/s), io=1428MiB '
'(1497MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_396.output'),
'state': {'begin': 1666783990.7907233,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.846900224685669,
'end': 1666783991.6376235,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_396.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_396.state')},
'cmd_40': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_40.output'),
'state': {'begin': 1666782232.8952932,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.8036515712738037,
'end': 1666782233.6989448,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_40.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_40.state')},
'cmd_41': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_41.output'),
'state': {'begin': 1666782233.7012796,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.568262815475464,
'end': 1666782245.2695425,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_41.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_41.state')},
'cmd_42': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789335,\n'
' "timestamp_ms" : '
'1666789335250,\n'
' "time" : "Wed Oct 26 15:02:15 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 483598848,\n'
' "io_kbytes" : 472264,\n'
' "bw_bytes" : 69075681,\n'
' "bw" : 67456,\n'
' "iops" : 134913.440937,\n'
' "runtime" : 7001,\n'
' "total_ios" : 944529,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 160,\n'
' "max" : 10830,\n'
' "mean" : 176.276544,\n'
' "stddev" : 42.021704,\n'
' "N" : 944529\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 6261,\n'
' "max" : 115436,\n'
' "mean" : 6989.813789,\n'
' "stddev" : '
'1151.700203,\n'
' "N" : 944529,\n'
' "percentile" : {\n'
' "1.000000" : 6496,\n'
' "5.000000" : 6688,\n'
' "10.000000" : 6752,\n'
' "20.000000" : 6752,\n'
' "30.000000" : 6752,\n'
' "40.000000" : 6816,\n'
' "50.000000" : 6816,\n'
' "60.000000" : 6880,\n'
' "70.000000" : 6880,\n'
' "80.000000" : 6944,\n'
' "90.000000" : 7008,\n'
' "95.000000" : 7136,\n'
' "99.000000" : 13120,\n'
' "99.500000" : 14656,\n'
' "99.900000" : 17024,\n'
' "99.950000" : 17280,\n'
' "99.990000" : 58624\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6432,\n'
' "max" : 115607,\n'
' "mean" : 7166.090333,\n'
' "stddev" : '
'1152.795263,\n'
' "N" : 944529\n'
' },\n'
' "bw_min" : 67337,\n'
' "bw_max" : 67822,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'67490.142857,\n'
' "bw_dev" : 143.619614,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 134674,\n'
' "iops_max" : 135645,\n'
' "iops_mean" : '
'134980.500000,\n'
' "iops_stddev" : '
'287.344272,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 54.442857,\n'
' "sys_cpu" : 45.528571,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 98.367546,\n'
' "20" : 1.614985,\n'
' "50" : 0.010000,\n'
' "100" : 0.013128,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=5597: Wed '
'Oct 26 15:02:15 2022\n'
' read: IOPS=135k, BW=65.9MiB/s '
'(69.1MB/s)(461MiB/7001msec)\n'
' slat (nsec): min=160, '
'max=10830, avg=176.28, '
'stdev=42.02\n'
' clat (usec): min=6, max=115, '
'avg= 6.99, stdev= 1.15\n'
' lat (usec): min=6, max=115, '
'avg= 7.17, stdev= 1.15\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6496], 5.00th=[ '
'6688], 10.00th=[ 6752], 20.00th=[ '
'6752],\n'
' | 30.00th=[ 6752], 40.00th=[ '
'6816], 50.00th=[ 6816], 60.00th=[ '
'6880],\n'
' | 70.00th=[ 6880], 80.00th=[ '
'6944], 90.00th=[ 7008], 95.00th=[ '
'7136],\n'
' | 99.00th=[13120], '
'99.50th=[14656], 99.90th=[17024], '
'99.95th=[17280],\n'
' | 99.99th=[58624]\n'
' bw ( KiB/s): min=67337, '
'max=67822, per=100.00%, '
'avg=67490.14, stdev=143.62, '
'samples=14\n'
' iops : min=134674, '
'max=135645, avg=134980.50, '
'stdev=287.34, samples=14\n'
' lat (usec) : 10=98.37%, '
'20=1.61%, 50=0.01%, 100=0.01%, '
'250=0.01%\n'
' cpu : usr=54.44%, '
'sys=45.53%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=944529,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=65.9MiB/s (69.1MB/s), '
'65.9MiB/s-65.9MiB/s '
'(69.1MB/s-69.1MB/s), io=461MiB '
'(484MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_42.output'),
'state': {'begin': 1666782245.2729244,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.8888406753540039,
'end': 1666782246.161765,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_42.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_42.state')},
'cmd_43': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_43.output'),
'state': {'begin': 1666782248.3104722,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.8213715553283691,
'end': 1666782249.1318438,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_43.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_43.state')},
'cmd_44': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_44.output'),
'state': {'begin': 1666782249.1338692,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.446779727935791,
'end': 1666782260.580649,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_44.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_44.state')},
'cmd_45': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789350,\n'
' "timestamp_ms" : '
'1666789350610,\n'
' "time" : "Wed Oct 26 15:02:30 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 482136064,\n'
' "io_kbytes" : 470836,\n'
' "bw_bytes" : 68866742,\n'
' "bw" : 67252,\n'
' "iops" : 134505.356378,\n'
' "runtime" : 7001,\n'
' "total_ios" : 941672,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 160,\n'
' "max" : 9859,\n'
' "mean" : 177.184497,\n'
' "stddev" : 34.333593,\n'
' "N" : 941672\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 6301,\n'
' "max" : 114083,\n'
' "mean" : 7024.269096,\n'
' "stddev" : '
'1168.593145,\n'
' "N" : 941672,\n'
' "percentile" : {\n'
' "1.000000" : 6560,\n'
' "5.000000" : 6752,\n'
' "10.000000" : 6752,\n'
' "20.000000" : 6816,\n'
' "30.000000" : 6816,\n'
' "40.000000" : 6816,\n'
' "50.000000" : 6880,\n'
' "60.000000" : 6880,\n'
' "70.000000" : 6944,\n'
' "80.000000" : 6944,\n'
' "90.000000" : 7072,\n'
' "95.000000" : 7136,\n'
' "99.000000" : 13120,\n'
' "99.500000" : 14656,\n'
' "99.900000" : 17024,\n'
' "99.950000" : 17280,\n'
' "99.990000" : 59136\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6472,\n'
' "max" : 114264,\n'
' "mean" : 7201.453593,\n'
' "stddev" : '
'1169.346311,\n'
' "N" : 941672\n'
' },\n'
' "bw_min" : 67168,\n'
' "bw_max" : 67432,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'67286.928571,\n'
' "bw_dev" : 82.405624,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 134336,\n'
' "iops_max" : 134865,\n'
' "iops_mean" : '
'134574.000000,\n'
' "iops_stddev" : '
'164.977854,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 58.614286,\n'
' "sys_cpu" : 41.357143,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 98.349638,\n'
' "20" : 1.631885,\n'
' "50" : 0.010000,\n'
' "100" : 0.014124,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=5654: Wed '
'Oct 26 15:02:30 2022\n'
' read: IOPS=135k, BW=65.7MiB/s '
'(68.9MB/s)(460MiB/7001msec)\n'
' slat (nsec): min=160, '
'max=9859, avg=177.18, '
'stdev=34.33\n'
' clat (usec): min=6, max=114, '
'avg= 7.02, stdev= 1.17\n'
' lat (usec): min=6, max=114, '
'avg= 7.20, stdev= 1.17\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6560], 5.00th=[ '
'6752], 10.00th=[ 6752], 20.00th=[ '
'6816],\n'
' | 30.00th=[ 6816], 40.00th=[ '
'6816], 50.00th=[ 6880], 60.00th=[ '
'6880],\n'
' | 70.00th=[ 6944], 80.00th=[ '
'6944], 90.00th=[ 7072], 95.00th=[ '
'7136],\n'
' | 99.00th=[13120], '
'99.50th=[14656], 99.90th=[17024], '
'99.95th=[17280],\n'
' | 99.99th=[59136]\n'
' bw ( KiB/s): min=67168, '
'max=67432, per=100.00%, '
'avg=67286.93, stdev=82.41, '
'samples=14\n'
' iops : min=134336, '
'max=134865, avg=134574.00, '
'stdev=164.98, samples=14\n'
' lat (usec) : 10=98.35%, '
'20=1.63%, 50=0.01%, 100=0.01%, '
'250=0.01%\n'
' cpu : usr=58.61%, '
'sys=41.36%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=941672,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=65.7MiB/s (68.9MB/s), '
'65.7MiB/s-65.7MiB/s '
'(68.9MB/s-68.9MB/s), io=460MiB '
'(482MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_45.output'),
'state': {'begin': 1666782260.583536,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.8505508899688721,
'end': 1666782261.4340868,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_45.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_45.state')},
'cmd_46': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_46.output'),
'state': {'begin': 1666782263.8989596,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.9638922214508057,
'end': 1666782264.8628519,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_46.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_46.state')},
'cmd_47': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_47.output'),
'state': {'begin': 1666782264.869519,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.649443626403809,
'end': 1666782276.5189626,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_47.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_47.state')},
'cmd_48': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789366,\n'
' "timestamp_ms" : '
'1666789366545,\n'
' "time" : "Wed Oct 26 15:02:46 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 551760384,\n'
' "io_kbytes" : 538828,\n'
' "bw_bytes" : 78811653,\n'
' "bw" : 76964,\n'
' "iops" : 153928.867305,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1077656,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1673,\n'
' "max" : 19667,\n'
' "mean" : 1817.265533,\n'
' "stddev" : 149.648202,\n'
' "N" : 1077657\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 240,\n'
' "max" : 99747,\n'
' "mean" : 4449.052741,\n'
' "stddev" : '
'1090.885033,\n'
' "N" : 1077656,\n'
' "percentile" : {\n'
' "1.000000" : 4016,\n'
' "5.000000" : 4192,\n'
' "10.000000" : 4256,\n'
' "20.000000" : 4256,\n'
' "30.000000" : 4256,\n'
' "40.000000" : 4320,\n'
' "50.000000" : 4320,\n'
' "60.000000" : 4320,\n'
' "70.000000" : 4384,\n'
' "80.000000" : 4384,\n'
' "90.000000" : 4448,\n'
' "95.000000" : 4512,\n'
' "99.000000" : 9664,\n'
' "99.500000" : 10688,\n'
' "99.900000" : 15296,\n'
' "99.950000" : 15424,\n'
' "99.990000" : 57088\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5521,\n'
' "max" : 101600,\n'
' "mean" : 6266.318333,\n'
' "stddev" : '
'1102.612816,\n'
' "N" : 1077656\n'
' },\n'
' "bw_min" : 76849,\n'
' "bw_max" : 77209,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'77006.785714,\n'
' "bw_dev" : 107.468907,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 153699,\n'
' "iops_max" : 154418,\n'
' "iops_mean" : '
'154013.714286,\n'
' "iops_stddev" : '
'214.803104,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 16.271429,\n'
' "sys_cpu" : 83.714286,\n'
' "ctx" : 7,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.958284,\n'
' "10" : 98.165184,\n'
' "20" : 0.857880,\n'
' "50" : 0.010000,\n'
' "100" : 0.012249,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=5712: Wed '
'Oct 26 15:02:46 2022\n'
' read: IOPS=154k, BW=75.2MiB/s '
'(78.8MB/s)(526MiB/7001msec)\n'
' slat (nsec): min=1673, '
'max=19667, avg=1817.27, '
'stdev=149.65\n'
' clat (nsec): min=240, '
'max=99747, avg=4449.05, '
'stdev=1090.89\n'
' lat (usec): min=5, max=101, '
'avg= 6.27, stdev= 1.10\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4016], 5.00th=[ '
'4192], 10.00th=[ 4256], 20.00th=[ '
'4256],\n'
' | 30.00th=[ 4256], 40.00th=[ '
'4320], 50.00th=[ 4320], 60.00th=[ '
'4320],\n'
' | 70.00th=[ 4384], 80.00th=[ '
'4384], 90.00th=[ 4448], 95.00th=[ '
'4512],\n'
' | 99.00th=[ 9664], '
'99.50th=[10688], 99.90th=[15296], '
'99.95th=[15424],\n'
' | 99.99th=[57088]\n'
' bw ( KiB/s): min=76849, '
'max=77209, per=100.00%, '
'avg=77006.79, stdev=107.47, '
'samples=14\n'
' iops : min=153699, '
'max=154418, avg=154013.71, '
'stdev=214.80, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%, 750=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.96%, 10=98.17%, 20=0.86%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%\n'
' cpu : usr=16.27%, '
'sys=83.71%, ctx=7, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1077656,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=75.2MiB/s (78.8MB/s), '
'75.2MiB/s-75.2MiB/s '
'(78.8MB/s-78.8MB/s), io=526MiB '
'(552MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_48.output'),
'state': {'begin': 1666782276.5220842,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8515095710754395,
'end': 1666782277.3735938,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_48.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_48.state')},
'cmd_49': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_49.output'),
'state': {'begin': 1666782279.6967657,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 1.0451750755310059,
'end': 1666782280.7419407,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_49.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_49.state')},
'cmd_50': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_50.output'),
'state': {'begin': 1666782280.7438326,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.476299524307251,
'end': 1666782292.220132,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_50.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_50.state')},
'cmd_51': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789382,\n'
' "timestamp_ms" : '
'1666789382237,\n'
' "time" : "Wed Oct 26 15:03:02 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 552491520,\n'
' "io_kbytes" : 539542,\n'
' "bw_bytes" : 78916086,\n'
' "bw" : 77066,\n'
' "iops" : 154132.981003,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1079085,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1643,\n'
' "max" : 21780,\n'
' "mean" : 1806.536104,\n'
' "stddev" : 149.212612,\n'
' "N" : 1079085\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 260,\n'
' "max" : 101110,\n'
' "mean" : 4453.978682,\n'
' "stddev" : '
'1094.494781,\n'
' "N" : 1079085,\n'
' "percentile" : {\n'
' "1.000000" : 4016,\n'
' "5.000000" : 4192,\n'
' "10.000000" : 4256,\n'
' "20.000000" : 4256,\n'
' "30.000000" : 4256,\n'
' "40.000000" : 4320,\n'
' "50.000000" : 4320,\n'
' "60.000000" : 4320,\n'
' "70.000000" : 4384,\n'
' "80.000000" : 4384,\n'
' "90.000000" : 4448,\n'
' "95.000000" : 4576,\n'
' "99.000000" : 9664,\n'
' "99.500000" : 10688,\n'
' "99.900000" : 15296,\n'
' "99.950000" : 15552,\n'
' "99.990000" : 57088\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5510,\n'
' "max" : 102933,\n'
' "mean" : 6260.514786,\n'
' "stddev" : '
'1106.115216,\n'
' "N" : 1079085\n'
' },\n'
' "bw_min" : 76941,\n'
' "bw_max" : 77411,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'77108.928571,\n'
' "bw_dev" : 145.888506,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 153883,\n'
' "iops_max" : 154822,\n'
' "iops_mean" : '
'154218.000000,\n'
' "iops_stddev" : '
'291.839524,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.014286,\n'
' "sys_cpu" : 82.971429,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.922634,\n'
' "10" : 98.195230,\n'
' "20" : 0.862768,\n'
' "50" : 0.010000,\n'
' "100" : 0.012140,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=5770: Wed '
'Oct 26 15:03:02 2022\n'
' read: IOPS=154k, BW=75.3MiB/s '
'(78.9MB/s)(527MiB/7001msec)\n'
' slat (nsec): min=1643, '
'max=21780, avg=1806.54, '
'stdev=149.21\n'
' clat (nsec): min=260, '
'max=101110, avg=4453.98, '
'stdev=1094.49\n'
' lat (usec): min=5, max=102, '
'avg= 6.26, stdev= 1.11\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4016], 5.00th=[ '
'4192], 10.00th=[ 4256], 20.00th=[ '
'4256],\n'
' | 30.00th=[ 4256], 40.00th=[ '
'4320], 50.00th=[ 4320], 60.00th=[ '
'4320],\n'
' | 70.00th=[ 4384], 80.00th=[ '
'4384], 90.00th=[ 4448], 95.00th=[ '
'4576],\n'
' | 99.00th=[ 9664], '
'99.50th=[10688], 99.90th=[15296], '
'99.95th=[15552],\n'
' | 99.99th=[57088]\n'
' bw ( KiB/s): min=76941, '
'max=77411, per=100.00%, '
'avg=77108.93, stdev=145.89, '
'samples=14\n'
' iops : min=153883, '
'max=154822, avg=154218.00, '
'stdev=291.84, samples=14\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%, 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.92%, 10=98.20%, 20=0.86%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%, '
'250=0.01%\n'
' cpu : usr=17.01%, '
'sys=82.97%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1079085,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=75.3MiB/s (78.9MB/s), '
'75.3MiB/s-75.3MiB/s '
'(78.9MB/s-78.9MB/s), io=527MiB '
'(552MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_51.output'),
'state': {'begin': 1666782292.2229006,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8048901557922363,
'end': 1666782293.0277908,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_51.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_51.state')},
'cmd_52': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_52.output'),
'state': {'begin': 1666782295.1316152,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8026421070098877,
'end': 1666782295.9342573,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_52.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_52.state')},
'cmd_53': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_53.output'),
'state': {'begin': 1666782295.9411457,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.441778421401978,
'end': 1666782307.382924,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_53.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_53.state')},
'cmd_54': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789397,\n'
' "timestamp_ms" : '
'1666789397412,\n'
' "time" : "Wed Oct 26 15:03:17 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 549262848,\n'
' "io_kbytes" : 536389,\n'
' "bw_bytes" : 78454913,\n'
' "bw" : 76616,\n'
' "iops" : 153232.252535,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1072779,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1663,\n'
' "max" : 17694,\n'
' "mean" : 1826.517569,\n'
' "stddev" : 127.585417,\n'
' "N" : 1072779\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 250,\n'
' "max" : 114004,\n'
' "mean" : 4474.261006,\n'
' "stddev" : '
'1097.446190,\n'
' "N" : 1072779,\n'
' "percentile" : {\n'
' "1.000000" : 4016,\n'
' "5.000000" : 4256,\n'
' "10.000000" : 4256,\n'
' "20.000000" : 4256,\n'
' "30.000000" : 4320,\n'
' "40.000000" : 4320,\n'
' "50.000000" : 4320,\n'
' "60.000000" : 4384,\n'
' "70.000000" : 4384,\n'
' "80.000000" : 4448,\n'
' "90.000000" : 4448,\n'
' "95.000000" : 4576,\n'
' "99.000000" : 9792,\n'
' "99.500000" : 10688,\n'
' "99.900000" : 15296,\n'
' "99.950000" : 15424,\n'
' "99.990000" : 57088\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5540,\n'
' "max" : 115927,\n'
' "mean" : 6300.778575,\n'
' "stddev" : '
'1106.768573,\n'
' "N" : 1072779\n'
' },\n'
' "bw_min" : 76414,\n'
' "bw_max" : 76744,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'76630.071429,\n'
' "bw_dev" : 96.866652,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 152828,\n'
' "iops_max" : 153488,\n'
' "iops_mean" : '
'153260.357143,\n'
' "iops_stddev" : '
'193.638683,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 21.700000,\n'
' "sys_cpu" : 78.285714,\n'
' "ctx" : 9,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.863552,\n'
' "10" : 98.266745,\n'
' "20" : 0.853577,\n'
' "50" : 0.010000,\n'
' "100" : 0.011466,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=5827: Wed '
'Oct 26 15:03:17 2022\n'
' read: IOPS=153k, BW=74.8MiB/s '
'(78.5MB/s)(524MiB/7001msec)\n'
' slat (nsec): min=1663, '
'max=17694, avg=1826.52, '
'stdev=127.59\n'
' clat (nsec): min=250, '
'max=114004, avg=4474.26, '
'stdev=1097.45\n'
' lat (usec): min=5, max=115, '
'avg= 6.30, stdev= 1.11\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4016], 5.00th=[ '
'4256], 10.00th=[ 4256], 20.00th=[ '
'4256],\n'
' | 30.00th=[ 4320], 40.00th=[ '
'4320], 50.00th=[ 4320], 60.00th=[ '
'4384],\n'
' | 70.00th=[ 4384], 80.00th=[ '
'4448], 90.00th=[ 4448], 95.00th=[ '
'4576],\n'
' | 99.00th=[ 9792], '
'99.50th=[10688], 99.90th=[15296], '
'99.95th=[15424],\n'
' | 99.99th=[57088]\n'
' bw ( KiB/s): min=76414, '
'max=76744, per=100.00%, '
'avg=76630.07, stdev=96.87, '
'samples=14\n'
' iops : min=152828, '
'max=153488, avg=153260.36, '
'stdev=193.64, samples=14\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%, 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.86%, 10=98.27%, 20=0.85%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%, '
'250=0.01%\n'
' cpu : usr=21.70%, '
'sys=78.29%, ctx=9, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1072779,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=74.8MiB/s (78.5MB/s), '
'74.8MiB/s-74.8MiB/s '
'(78.5MB/s-78.5MB/s), io=524MiB '
'(549MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_54.output'),
'state': {'begin': 1666782307.3854988,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8249804973602295,
'end': 1666782308.2104793,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_54.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_54.state')},
'cmd_55': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_55.output'),
'state': {'begin': 1666782309.2899835,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8343324661254883,
'end': 1666782310.124316,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_55.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_55.state')},
'cmd_56': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_56.output'),
'state': {'begin': 1666782310.1264532,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.075448036193848,
'end': 1666782321.2019012,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_56.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_56.state')},
'cmd_57': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789411,\n'
' "timestamp_ms" : '
'1666789411269,\n'
' "time" : "Wed Oct 26 15:03:31 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 390937600,\n'
' "io_kbytes" : 381775,\n'
' "bw_bytes" : 55840251,\n'
' "bw" : 54531,\n'
' "iops" : 109062.991001,\n'
' "runtime" : 7001,\n'
' "total_ios" : 763550,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1503,\n'
' "max" : 14387,\n'
' "mean" : 1667.139119,\n'
' "stddev" : 81.310916,\n'
' "N" : 763550\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 230,\n'
' "max" : 118001,\n'
' "mean" : 7204.880431,\n'
' "stddev" : 960.561063,\n'
' "N" : 763550,\n'
' "percentile" : {\n'
' "1.000000" : 6752,\n'
' "5.000000" : 6944,\n'
' "10.000000" : 6944,\n'
' "20.000000" : 7008,\n'
' "30.000000" : 7008,\n'
' "40.000000" : 7072,\n'
' "50.000000" : 7136,\n'
' "60.000000" : 7136,\n'
' "70.000000" : 7200,\n'
' "80.000000" : 7200,\n'
' "90.000000" : 7264,\n'
' "95.000000" : 7392,\n'
' "99.000000" : 10176,\n'
' "99.500000" : 13504,\n'
' "99.900000" : 15552,\n'
' "99.950000" : 15808,\n'
' "99.990000" : 57600\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7204,\n'
' "max" : 119654,\n'
' "mean" : 8872.019549,\n'
' "stddev" : 965.822097,\n'
' "N" : 763550\n'
' },\n'
' "bw_min" : 54161,\n'
' "bw_max" : 54881,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'54540.928571,\n'
' "bw_dev" : 272.108424,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 108322,\n'
' "iops_max" : 109762,\n'
' "iops_mean" : '
'109081.928571,\n'
' "iops_stddev" : '
'544.151628,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 8.357143,\n'
' "sys_cpu" : 36.128571,\n'
' "ctx" : 763553,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 98.860847,\n'
' "20" : 1.121865,\n'
' "50" : 0.010000,\n'
' "100" : 0.013228,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=5874: Wed '
'Oct 26 15:03:31 2022\n'
' read: IOPS=109k, BW=53.3MiB/s '
'(55.8MB/s)(373MiB/7001msec)\n'
' slat (nsec): min=1503, '
'max=14387, avg=1667.14, '
'stdev=81.31\n'
' clat (nsec): min=230, '
'max=118001, avg=7204.88, '
'stdev=960.56\n'
' lat (nsec): min=7204, '
'max=119654, avg=8872.02, '
'stdev=965.82\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6752], 5.00th=[ '
'6944], 10.00th=[ 6944], 20.00th=[ '
'7008],\n'
' | 30.00th=[ 7008], 40.00th=[ '
'7072], 50.00th=[ 7136], 60.00th=[ '
'7136],\n'
' | 70.00th=[ 7200], 80.00th=[ '
'7200], 90.00th=[ 7264], 95.00th=[ '
'7392],\n'
' | 99.00th=[10176], '
'99.50th=[13504], 99.90th=[15552], '
'99.95th=[15808],\n'
' | 99.99th=[57600]\n'
' bw ( KiB/s): min=54161, '
'max=54881, per=100.00%, '
'avg=54540.93, stdev=272.11, '
'samples=14\n'
' iops : min=108322, '
'max=109762, avg=109081.93, '
'stdev=544.15, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=98.86%, 20=1.12%, 50=0.01%, '
'100=0.01%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=8.36%, '
'sys=36.13%, ctx=763553, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=763550,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=53.3MiB/s (55.8MB/s), '
'53.3MiB/s-53.3MiB/s '
'(55.8MB/s-55.8MB/s), io=373MiB '
'(391MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_57.output'),
'state': {'begin': 1666782321.2053792,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8669404983520508,
'end': 1666782322.0723197,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_57.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_57.state')},
'cmd_58': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_58.output'),
'state': {'begin': 1666782323.139761,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8553993701934814,
'end': 1666782323.9951603,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_58.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_58.state')},
'cmd_59': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_59.output'),
'state': {'begin': 1666782323.9972491,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.102566242218018,
'end': 1666782335.0998154,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_59.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_59.state')},
'cmd_60': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789425,\n'
' "timestamp_ms" : '
'1666789425191,\n'
' "time" : "Wed Oct 26 15:03:45 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 390010880,\n'
' "io_kbytes" : 380870,\n'
' "bw_bytes" : 55715840,\n'
' "bw" : 54410,\n'
' "iops" : 108819.857143,\n'
' "runtime" : 7000,\n'
' "total_ios" : 761739,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1523,\n'
' "max" : 23895,\n'
' "mean" : 1676.788178,\n'
' "stddev" : 77.910712,\n'
' "N" : 761740\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 211,\n'
' "max" : 134522,\n'
' "mean" : 7217.694233,\n'
' "stddev" : '
'1010.392200,\n'
' "N" : 761739,\n'
' "percentile" : {\n'
' "1.000000" : 6752,\n'
' "5.000000" : 6944,\n'
' "10.000000" : 7008,\n'
' "20.000000" : 7008,\n'
' "30.000000" : 7008,\n'
' "40.000000" : 7072,\n'
' "50.000000" : 7136,\n'
' "60.000000" : 7136,\n'
' "70.000000" : 7200,\n'
' "80.000000" : 7200,\n'
' "90.000000" : 7328,\n'
' "95.000000" : 7392,\n'
' "99.000000" : 10176,\n'
' "99.500000" : 13632,\n'
' "99.900000" : 15552,\n'
' "99.950000" : 15808,\n'
' "99.990000" : 59648\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7003,\n'
' "max" : 136165,\n'
' "mean" : 8894.482495,\n'
' "stddev" : '
'1014.288753,\n'
' "N" : 761739\n'
' },\n'
' "bw_min" : 54271,\n'
' "bw_max" : 54567,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'54411.928571,\n'
' "bw_dev" : 78.668207,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 108542,\n'
' "iops_max" : 109134,\n'
' "iops_mean" : '
'108823.928571,\n'
' "iops_stddev" : '
'157.274119,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 9.200000,\n'
' "sys_cpu" : 35.271429,\n'
' "ctx" : 761738,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 98.872448,\n'
' "20" : 1.107466,\n'
' "50" : 0.010000,\n'
' "100" : 0.015228,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=5922: Wed '
'Oct 26 15:03:45 2022\n'
' read: IOPS=109k, BW=53.1MiB/s '
'(55.7MB/s)(372MiB/7000msec)\n'
' slat (nsec): min=1523, '
'max=23895, avg=1676.79, '
'stdev=77.91\n'
' clat (nsec): min=211, '
'max=134522, avg=7217.69, '
'stdev=1010.39\n'
' lat (usec): min=7, max=136, '
'avg= 8.89, stdev= 1.01\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6752], 5.00th=[ '
'6944], 10.00th=[ 7008], 20.00th=[ '
'7008],\n'
' | 30.00th=[ 7008], 40.00th=[ '
'7072], 50.00th=[ 7136], 60.00th=[ '
'7136],\n'
' | 70.00th=[ 7200], 80.00th=[ '
'7200], 90.00th=[ 7328], 95.00th=[ '
'7392],\n'
' | 99.00th=[10176], '
'99.50th=[13632], 99.90th=[15552], '
'99.95th=[15808],\n'
' | 99.99th=[59648]\n'
' bw ( KiB/s): min=54271, '
'max=54567, per=100.00%, '
'avg=54411.93, stdev=78.67, '
'samples=14\n'
' iops : min=108542, '
'max=109134, avg=108823.93, '
'stdev=157.27, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=98.87%, 20=1.11%, 50=0.01%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=9.20%, '
'sys=35.27%, ctx=761738, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=761739,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=53.1MiB/s (55.7MB/s), '
'53.1MiB/s-53.1MiB/s '
'(55.7MB/s-55.7MB/s), io=372MiB '
'(390MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_60.output'),
'state': {'begin': 1666782335.102811,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8670730590820312,
'end': 1666782335.9698842,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_60.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_60.state')},
'cmd_61': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_61.output'),
'state': {'begin': 1666782337.0434806,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8228695392608643,
'end': 1666782337.8663502,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_61.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_61.state')},
'cmd_62': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_62.output'),
'state': {'begin': 1666782337.8692353,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.072468042373657,
'end': 1666782348.9417033,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_62.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_62.state')},
'cmd_63': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789439,\n'
' "timestamp_ms" : '
'1666789439029,\n'
' "time" : "Wed Oct 26 15:03:59 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 390489088,\n'
' "io_kbytes" : 381337,\n'
' "bw_bytes" : 55776187,\n'
' "bw" : 54468,\n'
' "iops" : 108937.723182,\n'
' "runtime" : 7001,\n'
' "total_ios" : 762673,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1512,\n'
' "max" : 22973,\n'
' "mean" : 1663.808307,\n'
' "stddev" : 81.901888,\n'
' "N" : 762674\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2965,\n'
' "max" : 339156,\n'
' "mean" : 7211.033603,\n'
' "stddev" : '
'1048.223194,\n'
' "N" : 762673,\n'
' "percentile" : {\n'
' "1.000000" : 6752,\n'
' "5.000000" : 6944,\n'
' "10.000000" : 6944,\n'
' "20.000000" : 7008,\n'
' "30.000000" : 7008,\n'
' "40.000000" : 7072,\n'
' "50.000000" : 7136,\n'
' "60.000000" : 7136,\n'
' "70.000000" : 7200,\n'
' "80.000000" : 7200,\n'
' "90.000000" : 7264,\n'
' "95.000000" : 7392,\n'
' "99.000000" : 10176,\n'
' "99.500000" : 13504,\n'
' "99.900000" : 15552,\n'
' "99.950000" : 15808,\n'
' "99.990000" : 57600\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7304,\n'
' "max" : 340799,\n'
' "mean" : 8874.841846,\n'
' "stddev" : '
'1052.530746,\n'
' "N" : 762673\n'
' },\n'
' "bw_min" : 54216,\n'
' "bw_max" : 54781,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'54498.571429,\n'
' "bw_dev" : 162.638681,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 108432,\n'
' "iops_max" : 109562,\n'
' "iops_mean" : '
'108997.285714,\n'
' "iops_stddev" : '
'325.114283,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 10.714286,\n'
' "sys_cpu" : 33.800000,\n'
' "ctx" : 762675,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 98.851539,\n'
' "20" : 1.126957,\n'
' "50" : 0.010000,\n'
' "100" : 0.014685,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=5970: Wed '
'Oct 26 15:03:59 2022\n'
' read: IOPS=109k, BW=53.2MiB/s '
'(55.8MB/s)(372MiB/7001msec)\n'
' slat (nsec): min=1512, '
'max=22973, avg=1663.81, '
'stdev=81.90\n'
' clat (usec): min=2, max=339, '
'avg= 7.21, stdev= 1.05\n'
' lat (usec): min=7, max=340, '
'avg= 8.87, stdev= 1.05\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6752], 5.00th=[ '
'6944], 10.00th=[ 6944], 20.00th=[ '
'7008],\n'
' | 30.00th=[ 7008], 40.00th=[ '
'7072], 50.00th=[ 7136], 60.00th=[ '
'7136],\n'
' | 70.00th=[ 7200], 80.00th=[ '
'7200], 90.00th=[ 7264], 95.00th=[ '
'7392],\n'
' | 99.00th=[10176], '
'99.50th=[13504], 99.90th=[15552], '
'99.95th=[15808],\n'
' | 99.99th=[57600]\n'
' bw ( KiB/s): min=54216, '
'max=54781, per=100.00%, '
'avg=54498.57, stdev=162.64, '
'samples=14\n'
' iops : min=108432, '
'max=109562, avg=108997.29, '
'stdev=325.11, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=98.85%, 20=1.13%, 50=0.01%, '
'100=0.01%\n'
' lat (usec) : 500=0.01%\n'
' cpu : usr=10.71%, '
'sys=33.80%, ctx=762675, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=762673,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=53.2MiB/s (55.8MB/s), '
'53.2MiB/s-53.2MiB/s '
'(55.8MB/s-55.8MB/s), io=372MiB '
'(390MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_63.output'),
'state': {'begin': 1666782348.9444702,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7905919551849365,
'end': 1666782349.7350621,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_63.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_63.state')},
'cmd_64': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_libaio-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_64.output'),
'state': {'begin': 1666782350.802364,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8477296829223633,
'end': 1666782351.6500938,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_64.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_64.state')},
'cmd_65': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_65.output'),
'state': {'begin': 1666782351.652733,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.121761798858643,
'end': 1666782362.774495,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_65.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_65.state')},
'cmd_66': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789452,\n'
' "timestamp_ms" : '
'1666789452865,\n'
' "time" : "Wed Oct 26 15:04:12 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 391555584,\n'
' "io_kbytes" : 382378,\n'
' "bw_bytes" : 55928522,\n'
' "bw" : 54617,\n'
' "iops" : 109235.394944,\n'
' "runtime" : 7001,\n'
' "total_ios" : 764757,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1653,\n'
' "max" : 6341,\n'
' "mean" : 1761.576344,\n'
' "stddev" : 74.909216,\n'
' "N" : 764757\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2775,\n'
' "max" : 94186,\n'
' "mean" : 7098.314393,\n'
' "stddev" : 981.590607,\n'
' "N" : 764757,\n'
' "percentile" : {\n'
' "1.000000" : 6624,\n'
' "5.000000" : 6816,\n'
' "10.000000" : 6880,\n'
' "20.000000" : 6880,\n'
' "30.000000" : 6944,\n'
' "40.000000" : 6944,\n'
' "50.000000" : 7008,\n'
' "60.000000" : 7072,\n'
' "70.000000" : 7072,\n'
' "80.000000" : 7072,\n'
' "90.000000" : 7200,\n'
' "95.000000" : 7264,\n'
' "99.000000" : 10176,\n'
' "99.500000" : 13504,\n'
' "99.900000" : 15424,\n'
' "99.950000" : 15680,\n'
' "99.990000" : 58112\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7384,\n'
' "max" : 95980,\n'
' "mean" : 8859.890737,\n'
' "stddev" : 985.744709,\n'
' "N" : 764757\n'
' },\n'
' "bw_min" : 54489,\n'
' "bw_max" : 54838,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'54647.692308,\n'
' "bw_dev" : 106.501475,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 108978,\n'
' "iops_max" : 109676,\n'
' "iops_mean" : '
'109295.461538,\n'
' "iops_stddev" : '
'213.112887,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.628571,\n'
' "sys_cpu" : 27.585714,\n'
' "ctx" : 764759,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 98.872583,\n'
' "20" : 1.104534,\n'
' "50" : 0.010000,\n'
' "100" : 0.014253,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 1067722,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 5364,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 5364,\n'
' "util" : 99.045298\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=6017: Wed '
'Oct 26 15:04:12 2022\n'
' read: IOPS=109k, BW=53.3MiB/s '
'(55.9MB/s)(373MiB/7001msec)\n'
' slat (nsec): min=1653, '
'max=6341, avg=1761.58, '
'stdev=74.91\n'
' clat (nsec): min=2775, '
'max=94186, avg=7098.31, '
'stdev=981.59\n'
' lat (nsec): min=7384, '
'max=95980, avg=8859.89, '
'stdev=985.74\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6624], 5.00th=[ '
'6816], 10.00th=[ 6880], 20.00th=[ '
'6880],\n'
' | 30.00th=[ 6944], 40.00th=[ '
'6944], 50.00th=[ 7008], 60.00th=[ '
'7072],\n'
' | 70.00th=[ 7072], 80.00th=[ '
'7072], 90.00th=[ 7200], 95.00th=[ '
'7264],\n'
' | 99.00th=[10176], '
'99.50th=[13504], 99.90th=[15424], '
'99.95th=[15680],\n'
' | 99.99th=[58112]\n'
' bw ( KiB/s): min=54489, '
'max=54838, per=100.00%, '
'avg=54647.69, stdev=106.50, '
'samples=13\n'
' iops : min=108978, '
'max=109676, avg=109295.46, '
'stdev=213.11, samples=13\n'
' lat (usec) : 4=0.01%, '
'10=98.87%, 20=1.10%, 50=0.01%, '
'100=0.01%\n'
' cpu : usr=15.63%, '
'sys=27.59%, ctx=764759, majf=0, '
'minf=1\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=764757,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=53.3MiB/s (55.9MB/s), '
'53.3MiB/s-53.3MiB/s '
'(55.9MB/s-55.9MB/s), io=373MiB '
'(392MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=1067722/0, '
'merge=0/0, ticks=5364/0, '
'in_queue=5364, util=99.05%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_66.output'),
'state': {'begin': 1666782362.77719,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8729548454284668,
'end': 1666782363.6501448,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_66.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_66.state')},
'cmd_67': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_libaio-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_67.output'),
'state': {'begin': 1666782364.7075977,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7930121421813965,
'end': 1666782365.5006099,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_67.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_67.state')},
'cmd_68': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_68.output'),
'state': {'begin': 1666782365.5025218,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.134260654449463,
'end': 1666782376.6367824,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_68.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_68.state')},
'cmd_69': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789466,\n'
' "timestamp_ms" : '
'1666789466728,\n'
' "time" : "Wed Oct 26 15:04:26 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 393853440,\n'
' "io_kbytes" : 384622,\n'
' "bw_bytes" : 56256740,\n'
' "bw" : 54938,\n'
' "iops" : 109876.446222,\n'
' "runtime" : 7001,\n'
' "total_ios" : 769245,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1633,\n'
' "max" : 15660,\n'
' "mean" : 1733.714902,\n'
' "stddev" : 59.717360,\n'
' "N" : 769245\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 581,\n'
' "max" : 88846,\n'
' "mean" : 7084.517244,\n'
' "stddev" : 968.322335,\n'
' "N" : 769245,\n'
' "percentile" : {\n'
' "1.000000" : 6624,\n'
' "5.000000" : 6816,\n'
' "10.000000" : 6816,\n'
' "20.000000" : 6880,\n'
' "30.000000" : 6880,\n'
' "40.000000" : 6944,\n'
' "50.000000" : 7008,\n'
' "60.000000" : 7008,\n'
' "70.000000" : 7072,\n'
' "80.000000" : 7072,\n'
' "90.000000" : 7200,\n'
' "95.000000" : 7264,\n'
' "99.000000" : 10176,\n'
' "99.500000" : 13504,\n'
' "99.900000" : 15424,\n'
' "99.950000" : 15680,\n'
' "99.990000" : 58112\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7294,\n'
' "max" : 90539,\n'
' "mean" : 8818.232146,\n'
' "stddev" : 971.397702,\n'
' "N" : 769245\n'
' },\n'
' "bw_min" : 54727,\n'
' "bw_max" : 55130,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'54952.357143,\n'
' "bw_dev" : 130.746147,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 109454,\n'
' "iops_max" : 110260,\n'
' "iops_mean" : '
'109904.714286,\n'
' "iops_stddev" : '
'261.492294,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.442857,\n'
' "sys_cpu" : 25.171429,\n'
' "ctx" : 769247,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 98.809872,\n'
' "20" : 1.172708,\n'
' "50" : 0.010000,\n'
' "100" : 0.013650,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 1099355,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 5515,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 5515,\n'
' "util" : 99.069123\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=6063: Wed '
'Oct 26 15:04:26 2022\n'
' read: IOPS=110k, BW=53.7MiB/s '
'(56.3MB/s)(376MiB/7001msec)\n'
' slat (nsec): min=1633, '
'max=15660, avg=1733.71, '
'stdev=59.72\n'
' clat (nsec): min=581, '
'max=88846, avg=7084.52, '
'stdev=968.32\n'
' lat (nsec): min=7294, '
'max=90539, avg=8818.23, '
'stdev=971.40\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6624], 5.00th=[ '
'6816], 10.00th=[ 6816], 20.00th=[ '
'6880],\n'
' | 30.00th=[ 6880], 40.00th=[ '
'6944], 50.00th=[ 7008], 60.00th=[ '
'7008],\n'
' | 70.00th=[ 7072], 80.00th=[ '
'7072], 90.00th=[ 7200], 95.00th=[ '
'7264],\n'
' | 99.00th=[10176], '
'99.50th=[13504], 99.90th=[15424], '
'99.95th=[15680],\n'
' | 99.99th=[58112]\n'
' bw ( KiB/s): min=54727, '
'max=55130, per=100.00%, '
'avg=54952.36, stdev=130.75, '
'samples=14\n'
' iops : min=109454, '
'max=110260, avg=109904.71, '
'stdev=261.49, samples=14\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=98.81%, 20=1.17%, 50=0.01%, '
'100=0.01%\n'
' cpu : usr=17.44%, '
'sys=25.17%, ctx=769247, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=769245,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=53.7MiB/s (56.3MB/s), '
'53.7MiB/s-53.7MiB/s '
'(56.3MB/s-56.3MB/s), io=376MiB '
'(394MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=1099355/0, '
'merge=0/0, ticks=5515/0, '
'in_queue=5515, util=99.07%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_69.output'),
'state': {'begin': 1666782376.6392207,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8122994899749756,
'end': 1666782377.4515202,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_69.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_69.state')},
'cmd_70': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_libaio-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_70.output'),
'state': {'begin': 1666782378.5656462,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.8047864437103271,
'end': 1666782379.3704326,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_70.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_70.state')},
'cmd_71': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_71.output'),
'state': {'begin': 1666782379.3727932,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.141789436340332,
'end': 1666782390.5145826,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_71.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_71.state')},
'cmd_72': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789480,\n'
' "timestamp_ms" : '
'1666789480604,\n'
' "time" : "Wed Oct 26 15:04:40 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 392696320,\n'
' "io_kbytes" : 383492,\n'
' "bw_bytes" : 56099474,\n'
' "bw" : 54784,\n'
' "iops" : 109569.285714,\n'
' "runtime" : 7000,\n'
' "total_ios" : 766985,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1613,\n'
' "max" : 9728,\n'
' "mean" : 1717.578988,\n'
' "stddev" : 59.288933,\n'
' "N" : 766985\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 471,\n'
' "max" : 114094,\n'
' "mean" : 7108.687395,\n'
' "stddev" : '
'1019.143377,\n'
' "N" : 766985,\n'
' "percentile" : {\n'
' "1.000000" : 6624,\n'
' "5.000000" : 6816,\n'
' "10.000000" : 6880,\n'
' "20.000000" : 6880,\n'
' "30.000000" : 6944,\n'
' "40.000000" : 6944,\n'
' "50.000000" : 7008,\n'
' "60.000000" : 7072,\n'
' "70.000000" : 7072,\n'
' "80.000000" : 7136,\n'
' "90.000000" : 7200,\n'
' "95.000000" : 7328,\n'
' "99.000000" : 10176,\n'
' "99.500000" : 13504,\n'
' "99.900000" : 15552,\n'
' "99.950000" : 15680,\n'
' "99.990000" : 60160\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6893,\n'
' "max" : 115777,\n'
' "mean" : 8826.266383,\n'
' "stddev" : '
'1021.839338,\n'
' "N" : 766985\n'
' },\n'
' "bw_min" : 54634,\n'
' "bw_max" : 55015,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'54791.000000,\n'
' "bw_dev" : 131.780704,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 109268,\n'
' "iops_max" : 110030,\n'
' "iops_mean" : '
'109582.071429,\n'
' "iops_stddev" : '
'263.670092,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 19.400000,\n'
' "sys_cpu" : 23.571429,\n'
' "ctx" : 766986,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 98.838439,\n'
' "20" : 1.141222,\n'
' "50" : 0.010000,\n'
' "100" : 0.016037,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 1099551,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 5504,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 5504,\n'
' "util" : 99.069123\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=6110: Wed '
'Oct 26 15:04:40 2022\n'
' read: IOPS=110k, BW=53.5MiB/s '
'(56.1MB/s)(375MiB/7000msec)\n'
' slat (nsec): min=1613, '
'max=9728, avg=1717.58, '
'stdev=59.29\n'
' clat (nsec): min=471, '
'max=114094, avg=7108.69, '
'stdev=1019.14\n'
' lat (usec): min=6, max=115, '
'avg= 8.83, stdev= 1.02\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6624], 5.00th=[ '
'6816], 10.00th=[ 6880], 20.00th=[ '
'6880],\n'
' | 30.00th=[ 6944], 40.00th=[ '
'6944], 50.00th=[ 7008], 60.00th=[ '
'7072],\n'
' | 70.00th=[ 7072], 80.00th=[ '
'7136], 90.00th=[ 7200], 95.00th=[ '
'7328],\n'
' | 99.00th=[10176], '
'99.50th=[13504], 99.90th=[15552], '
'99.95th=[15680],\n'
' | 99.99th=[60160]\n'
' bw ( KiB/s): min=54634, '
'max=55015, per=100.00%, '
'avg=54791.00, stdev=131.78, '
'samples=14\n'
' iops : min=109268, '
'max=110030, avg=109582.07, '
'stdev=263.67, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=98.84%, 20=1.14%, 50=0.01%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=19.40%, '
'sys=23.57%, ctx=766986, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=766985,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=53.5MiB/s (56.1MB/s), '
'53.5MiB/s-53.5MiB/s '
'(56.1MB/s-56.1MB/s), io=375MiB '
'(393MB), run=7000-7000msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=1099551/0, '
'merge=0/0, ticks=5504/0, '
'in_queue=5504, util=99.07%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_72.output'),
'state': {'begin': 1666782390.517187,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.8540084362030029,
'end': 1666782391.3711956,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_72.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_72.state')},
'cmd_73': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_libaio-bdev_aio_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_73.output'),
'state': {'begin': 1666782393.6421034,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.8411805629730225,
'end': 1666782394.483284,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_73.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_73.state')},
'cmd_74': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_74.output'),
'state': {'begin': 1666782394.4855397,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-bdev_aio_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.499650239944458,
'end': 1666782405.98519,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_74.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_74.state')},
'cmd_75': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789495,\n'
' "timestamp_ms" : '
'1666789495981,\n'
' "time" : "Wed Oct 26 15:04:55 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 542872576,\n'
' "io_kbytes" : 530149,\n'
' "bw_bytes" : 77553225,\n'
' "bw" : 75735,\n'
' "iops" : 151471.000000,\n'
' "runtime" : 7000,\n'
' "total_ios" : 1060297,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1733,\n'
' "max" : 17563,\n'
' "mean" : 1907.857454,\n'
' "stddev" : 129.896684,\n'
' "N" : 1060298\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 230,\n'
' "max" : 96982,\n'
' "mean" : 4470.611756,\n'
' "stddev" : '
'1041.750438,\n'
' "N" : 1060297,\n'
' "percentile" : {\n'
' "1.000000" : 4048,\n'
' "5.000000" : 4256,\n'
' "10.000000" : 4256,\n'
' "20.000000" : 4256,\n'
' "30.000000" : 4320,\n'
' "40.000000" : 4320,\n'
' "50.000000" : 4320,\n'
' "60.000000" : 4384,\n'
' "70.000000" : 4384,\n'
' "80.000000" : 4448,\n'
' "90.000000" : 4448,\n'
' "95.000000" : 4640,\n'
' "99.000000" : 9664,\n'
' "99.500000" : 10688,\n'
' "99.900000" : 15296,\n'
' "99.950000" : 15424,\n'
' "99.990000" : 49408\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5640,\n'
' "max" : 98935,\n'
' "mean" : 6378.469139,\n'
' "stddev" : '
'1051.660338,\n'
' "N" : 1060297\n'
' },\n'
' "bw_min" : 75451,\n'
' "bw_max" : 76128,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'75738.142857,\n'
' "bw_dev" : 208.255707,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 150902,\n'
' "iops_max" : 152256,\n'
' "iops_mean" : '
'151476.285714,\n'
' "iops_stddev" : '
'416.511414,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 77.700000,\n'
' "sys_cpu" : 22.271429,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.755637,\n'
' "10" : 98.431383,\n'
' "20" : 0.798361,\n'
' "50" : 0.010000,\n'
' "100" : 0.010000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=6168: Wed '
'Oct 26 15:04:55 2022\n'
' read: IOPS=151k, BW=74.0MiB/s '
'(77.6MB/s)(518MiB/7000msec)\n'
' slat (nsec): min=1733, '
'max=17563, avg=1907.86, '
'stdev=129.90\n'
' clat (nsec): min=230, '
'max=96982, avg=4470.61, '
'stdev=1041.75\n'
' lat (nsec): min=5640, '
'max=98935, avg=6378.47, '
'stdev=1051.66\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4048], 5.00th=[ '
'4256], 10.00th=[ 4256], 20.00th=[ '
'4256],\n'
' | 30.00th=[ 4320], 40.00th=[ '
'4320], 50.00th=[ 4320], 60.00th=[ '
'4384],\n'
' | 70.00th=[ 4384], 80.00th=[ '
'4448], 90.00th=[ 4448], 95.00th=[ '
'4640],\n'
' | 99.00th=[ 9664], '
'99.50th=[10688], 99.90th=[15296], '
'99.95th=[15424],\n'
' | 99.99th=[49408]\n'
' bw ( KiB/s): min=75451, '
'max=76128, per=100.00%, '
'avg=75738.14, stdev=208.26, '
'samples=14\n'
' iops : min=150902, '
'max=152256, avg=151476.29, '
'stdev=416.51, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.76%, 10=98.43%, 20=0.80%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%\n'
' cpu : usr=77.70%, '
'sys=22.27%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1060297,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=74.0MiB/s (77.6MB/s), '
'74.0MiB/s-74.0MiB/s '
'(77.6MB/s-77.6MB/s), io=518MiB '
'(543MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_75.output'),
'state': {'begin': 1666782405.9877403,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.8320722579956055,
'end': 1666782406.8198125,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_75.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_75.state')},
'cmd_76': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_libaio-bdev_aio_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_76.output'),
'state': {'begin': 1666782408.9964058,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7969005107879639,
'end': 1666782409.7933064,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_76.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_76.state')},
'cmd_77': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_77.output'),
'state': {'begin': 1666782409.7953362,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-bdev_aio_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.568880081176758,
'end': 1666782421.3642163,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_77.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_77.state')},
'cmd_78': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789511,\n'
' "timestamp_ms" : '
'1666789511342,\n'
' "time" : "Wed Oct 26 15:05:11 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 544499200,\n'
' "io_kbytes" : 531737,\n'
' "bw_bytes" : 77774489,\n'
' "bw" : 75951,\n'
' "iops" : 151903.299529,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1063475,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1753,\n'
' "max" : 16831,\n'
' "mean" : 1900.398060,\n'
' "stddev" : 139.445188,\n'
' "N" : 1063475\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 210,\n'
' "max" : 77395,\n'
' "mean" : 4459.766281,\n'
' "stddev" : '
'1077.834294,\n'
' "N" : 1063475,\n'
' "percentile" : {\n'
' "1.000000" : 4048,\n'
' "5.000000" : 4192,\n'
' "10.000000" : 4256,\n'
' "20.000000" : 4256,\n'
' "30.000000" : 4256,\n'
' "40.000000" : 4320,\n'
' "50.000000" : 4320,\n'
' "60.000000" : 4384,\n'
' "70.000000" : 4384,\n'
' "80.000000" : 4384,\n'
' "90.000000" : 4448,\n'
' "95.000000" : 4576,\n'
' "99.000000" : 9664,\n'
' "99.500000" : 10560,\n'
' "99.900000" : 15296,\n'
' "99.950000" : 15424,\n'
' "99.990000" : 57088\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5620,\n'
' "max" : 79289,\n'
' "mean" : 6360.164341,\n'
' "stddev" : '
'1089.019246,\n'
' "N" : 1063475\n'
' },\n'
' "bw_min" : 75781,\n'
' "bw_max" : 76168,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'75991.692308,\n'
' "bw_dev" : 122.716058,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 151562,\n'
' "iops_max" : 152336,\n'
' "iops_mean" : '
'151983.461538,\n'
' "iops_stddev" : '
'245.379507,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 74.900000,\n'
' "sys_cpu" : 25.071429,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.799643,\n'
' "10" : 98.374292,\n'
' "20" : 0.809046,\n'
' "50" : 0.010000,\n'
' "100" : 0.012130,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=6226: Wed '
'Oct 26 15:05:11 2022\n'
' read: IOPS=152k, BW=74.2MiB/s '
'(77.8MB/s)(519MiB/7001msec)\n'
' slat (nsec): min=1753, '
'max=16831, avg=1900.40, '
'stdev=139.45\n'
' clat (nsec): min=210, '
'max=77395, avg=4459.77, '
'stdev=1077.83\n'
' lat (nsec): min=5620, '
'max=79289, avg=6360.16, '
'stdev=1089.02\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4048], 5.00th=[ '
'4192], 10.00th=[ 4256], 20.00th=[ '
'4256],\n'
' | 30.00th=[ 4256], 40.00th=[ '
'4320], 50.00th=[ 4320], 60.00th=[ '
'4384],\n'
' | 70.00th=[ 4384], 80.00th=[ '
'4384], 90.00th=[ 4448], 95.00th=[ '
'4576],\n'
' | 99.00th=[ 9664], '
'99.50th=[10560], 99.90th=[15296], '
'99.95th=[15424],\n'
' | 99.99th=[57088]\n'
' bw ( KiB/s): min=75781, '
'max=76168, per=100.00%, '
'avg=75991.69, stdev=122.72, '
'samples=13\n'
' iops : min=151562, '
'max=152336, avg=151983.46, '
'stdev=245.38, samples=13\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%, 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.80%, 10=98.37%, 20=0.81%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%\n'
' cpu : usr=74.90%, '
'sys=25.07%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1063475,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=74.2MiB/s (77.8MB/s), '
'74.2MiB/s-74.2MiB/s '
'(77.8MB/s-77.8MB/s), io=519MiB '
'(544MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_78.output'),
'state': {'begin': 1666782421.3675125,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.8112480640411377,
'end': 1666782422.1787605,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_78.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_78.state')},
'cmd_79': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_libaio-bdev_aio_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_79.output'),
'state': {'begin': 1666782424.267234,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.9013993740081787,
'end': 1666782425.1686335,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_79.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_79.state')},
'cmd_80': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_80.output'),
'state': {'begin': 1666782425.1756573,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-bdev_aio_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.547270059585571,
'end': 1666782436.7229273,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_80.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_80.state')},
'cmd_81': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789526,\n'
' "timestamp_ms" : '
'1666789526714,\n'
' "time" : "Wed Oct 26 15:05:26 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 542993920,\n'
' "io_kbytes" : 530267,\n'
' "bw_bytes" : 77559480,\n'
' "bw" : 75741,\n'
' "iops" : 151483.359520,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1060535,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1733,\n'
' "max" : 17993,\n'
' "mean" : 1902.296994,\n'
' "stddev" : 142.855709,\n'
' "N" : 1060535\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 220,\n'
' "max" : 99156,\n'
' "mean" : 4465.624404,\n'
' "stddev" : '
'1078.405463,\n'
' "N" : 1060535,\n'
' "percentile" : {\n'
' "1.000000" : 4048,\n'
' "5.000000" : 4256,\n'
' "10.000000" : 4256,\n'
' "20.000000" : 4256,\n'
' "30.000000" : 4320,\n'
' "40.000000" : 4320,\n'
' "50.000000" : 4320,\n'
' "60.000000" : 4384,\n'
' "70.000000" : 4384,\n'
' "80.000000" : 4384,\n'
' "90.000000" : 4448,\n'
' "95.000000" : 4576,\n'
' "99.000000" : 9664,\n'
' "99.500000" : 10688,\n'
' "99.900000" : 15296,\n'
' "99.950000" : 15424,\n'
' "99.990000" : 57088\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5630,\n'
' "max" : 101070,\n'
' "mean" : 6367.921398,\n'
' "stddev" : '
'1089.091086,\n'
' "N" : 1060535\n'
' },\n'
' "bw_min" : 75607,\n'
' "bw_max" : 75920,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'75783.928571,\n'
' "bw_dev" : 71.611628,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 151214,\n'
' "iops_max" : 151838,\n'
' "iops_mean" : '
'151567.857143,\n'
' "iops_stddev" : '
'142.936703,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 76.042857,\n'
' "sys_cpu" : 23.928571,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.758202,\n'
' "10" : 98.419666,\n'
' "20" : 0.804028,\n'
' "50" : 0.010000,\n'
' "100" : 0.011881,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=6285: Wed '
'Oct 26 15:05:26 2022\n'
' read: IOPS=151k, BW=74.0MiB/s '
'(77.6MB/s)(518MiB/7001msec)\n'
' slat (nsec): min=1733, '
'max=17993, avg=1902.30, '
'stdev=142.86\n'
' clat (nsec): min=220, '
'max=99156, avg=4465.62, '
'stdev=1078.41\n'
' lat (usec): min=5, max=101, '
'avg= 6.37, stdev= 1.09\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4048], 5.00th=[ '
'4256], 10.00th=[ 4256], 20.00th=[ '
'4256],\n'
' | 30.00th=[ 4320], 40.00th=[ '
'4320], 50.00th=[ 4320], 60.00th=[ '
'4384],\n'
' | 70.00th=[ 4384], 80.00th=[ '
'4384], 90.00th=[ 4448], 95.00th=[ '
'4576],\n'
' | 99.00th=[ 9664], '
'99.50th=[10688], 99.90th=[15296], '
'99.95th=[15424],\n'
' | 99.99th=[57088]\n'
' bw ( KiB/s): min=75607, '
'max=75920, per=100.00%, '
'avg=75783.93, stdev=71.61, '
'samples=14\n'
' iops : min=151214, '
'max=151838, avg=151567.86, '
'stdev=142.94, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%, 750=0.01%, 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.76%, 10=98.42%, 20=0.80%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%\n'
' cpu : usr=76.04%, '
'sys=23.93%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1060535,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=74.0MiB/s (77.6MB/s), '
'74.0MiB/s-74.0MiB/s '
'(77.6MB/s-77.6MB/s), io=518MiB '
'(543MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_81.output'),
'state': {'begin': 1666782436.729881,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.813136100769043,
'end': 1666782437.5430171,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_81.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_81.state')},
'cmd_82': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_82.output'),
'state': {'begin': 1666782439.7651637,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8066737651824951,
'end': 1666782440.5718374,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_82.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_82.state')},
'cmd_83': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_83.output'),
'state': {'begin': 1666782440.5739655,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.549588680267334,
'end': 1666782452.1235542,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_83.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_83.state')},
'cmd_84': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789542,\n'
' "timestamp_ms" : '
'1666789542121,\n'
' "time" : "Wed Oct 26 15:05:42 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 523051520,\n'
' "io_kbytes" : 510792,\n'
' "bw_bytes" : 74710972,\n'
' "bw" : 72959,\n'
' "iops" : 145919.868590,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1021585,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1753,\n'
' "max" : 40205,\n'
' "mean" : 1902.474167,\n'
' "stddev" : 112.382836,\n'
' "N" : 1021585\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 591,\n'
' "max" : 101370,\n'
' "mean" : 4716.904850,\n'
' "stddev" : '
'1098.648880,\n'
' "N" : 1021585,\n'
' "percentile" : {\n'
' "1.000000" : 4384,\n'
' "5.000000" : 4384,\n'
' "10.000000" : 4448,\n'
' "20.000000" : 4448,\n'
' "30.000000" : 4448,\n'
' "40.000000" : 4640,\n'
' "50.000000" : 4640,\n'
' "60.000000" : 4640,\n'
' "70.000000" : 4704,\n'
' "80.000000" : 4704,\n'
' "90.000000" : 4704,\n'
' "95.000000" : 4896,\n'
' "99.000000" : 9664,\n'
' "99.500000" : 10944,\n'
' "99.900000" : 15296,\n'
' "99.950000" : 15424,\n'
' "99.990000" : 58624\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5871,\n'
' "max" : 103304,\n'
' "mean" : 6619.379017,\n'
' "stddev" : '
'1103.785829,\n'
' "N" : 1021585\n'
' },\n'
' "bw_min" : 72756,\n'
' "bw_max" : 73193,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'72972.928571,\n'
' "bw_dev" : 114.125750,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 145512,\n'
' "iops_max" : 146386,\n'
' "iops_mean" : '
'145945.857143,\n'
' "iops_stddev" : '
'228.251500,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 54.728571,\n'
' "sys_cpu" : 45.242857,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.017913,\n'
' "10" : 99.114122,\n'
' "20" : 0.844962,\n'
' "50" : 0.010000,\n'
' "100" : 0.014389,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=6343: Wed '
'Oct 26 15:05:42 2022\n'
' read: IOPS=146k, BW=71.2MiB/s '
'(74.7MB/s)(499MiB/7001msec)\n'
' slat (nsec): min=1753, '
'max=40205, avg=1902.47, '
'stdev=112.38\n'
' clat (nsec): min=591, '
'max=101370, avg=4716.90, '
'stdev=1098.65\n'
' lat (usec): min=5, max=103, '
'avg= 6.62, stdev= 1.10\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4384], 5.00th=[ '
'4384], 10.00th=[ 4448], 20.00th=[ '
'4448],\n'
' | 30.00th=[ 4448], 40.00th=[ '
'4640], 50.00th=[ 4640], 60.00th=[ '
'4640],\n'
' | 70.00th=[ 4704], 80.00th=[ '
'4704], 90.00th=[ 4704], 95.00th=[ '
'4896],\n'
' | 99.00th=[ 9664], '
'99.50th=[10944], 99.90th=[15296], '
'99.95th=[15424],\n'
' | 99.99th=[58624]\n'
' bw ( KiB/s): min=72756, '
'max=73193, per=100.00%, '
'avg=72972.93, stdev=114.13, '
'samples=14\n'
' iops : min=145512, '
'max=146386, avg=145945.86, '
'stdev=228.25, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.02%, 10=99.11%, 20=0.84%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%, '
'250=0.01%\n'
' cpu : usr=54.73%, '
'sys=45.24%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1021585,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=71.2MiB/s (74.7MB/s), '
'71.2MiB/s-71.2MiB/s '
'(74.7MB/s-74.7MB/s), io=499MiB '
'(523MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_84.output'),
'state': {'begin': 1666782452.1262112,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7958531379699707,
'end': 1666782452.9220643,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_84.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_84.state')},
'cmd_85': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_85.output'),
'state': {'begin': 1666782455.0286593,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7907915115356445,
'end': 1666782455.8194509,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_85.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_85.state')},
'cmd_86': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_86.output'),
'state': {'begin': 1666782455.826383,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.405316829681396,
'end': 1666782467.2317,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_86.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_86.state')},
'cmd_87': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789557,\n'
' "timestamp_ms" : '
'1666789557225,\n'
' "time" : "Wed Oct 26 15:05:57 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 522268672,\n'
' "io_kbytes" : 510028,\n'
' "bw_bytes" : 74609810,\n'
' "bw" : 72861,\n'
' "iops" : 145722.142857,\n'
' "runtime" : 7000,\n'
' "total_ios" : 1020055,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1773,\n'
' "max" : 14247,\n'
' "mean" : 1911.317889,\n'
' "stddev" : 104.278118,\n'
' "N" : 1020056\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 601,\n'
' "max" : 117801,\n'
' "mean" : 4724.615978,\n'
' "stddev" : '
'1066.997954,\n'
' "N" : 1020055,\n'
' "percentile" : {\n'
' "1.000000" : 4256,\n'
' "5.000000" : 4448,\n'
' "10.000000" : 4448,\n'
' "20.000000" : 4448,\n'
' "30.000000" : 4512,\n'
' "40.000000" : 4512,\n'
' "50.000000" : 4640,\n'
' "60.000000" : 4704,\n'
' "70.000000" : 4704,\n'
' "80.000000" : 4704,\n'
' "90.000000" : 4768,\n'
' "95.000000" : 4768,\n'
' "99.000000" : 9664,\n'
' "99.500000" : 10944,\n'
' "99.900000" : 15296,\n'
' "99.950000" : 15424,\n'
' "99.990000" : 57088\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5851,\n'
' "max" : 119655,\n'
' "mean" : 6635.933895,\n'
' "stddev" : '
'1071.288496,\n'
' "N" : 1020055\n'
' },\n'
' "bw_min" : 72757,\n'
' "bw_max" : 73024,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'72863.785714,\n'
' "bw_dev" : 78.671839,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 145514,\n'
' "iops_max" : 146048,\n'
' "iops_mean" : '
'145727.642857,\n'
' "iops_stddev" : '
'157.314779,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 48.328571,\n'
' "sys_cpu" : 51.642857,\n'
' "ctx" : 14,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.014607,\n'
' "10" : 99.142791,\n'
' "20" : 0.821034,\n'
' "50" : 0.010000,\n'
' "100" : 0.012646,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=6400: Wed '
'Oct 26 15:05:57 2022\n'
' read: IOPS=146k, BW=71.2MiB/s '
'(74.6MB/s)(498MiB/7000msec)\n'
' slat (nsec): min=1773, '
'max=14247, avg=1911.32, '
'stdev=104.28\n'
' clat (nsec): min=601, '
'max=117801, avg=4724.62, '
'stdev=1067.00\n'
' lat (usec): min=5, max=119, '
'avg= 6.64, stdev= 1.07\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4256], 5.00th=[ '
'4448], 10.00th=[ 4448], 20.00th=[ '
'4448],\n'
' | 30.00th=[ 4512], 40.00th=[ '
'4512], 50.00th=[ 4640], 60.00th=[ '
'4704],\n'
' | 70.00th=[ 4704], 80.00th=[ '
'4704], 90.00th=[ 4768], 95.00th=[ '
'4768],\n'
' | 99.00th=[ 9664], '
'99.50th=[10944], 99.90th=[15296], '
'99.95th=[15424],\n'
' | 99.99th=[57088]\n'
' bw ( KiB/s): min=72757, '
'max=73024, per=100.00%, '
'avg=72863.79, stdev=78.67, '
'samples=14\n'
' iops : min=145514, '
'max=146048, avg=145727.64, '
'stdev=157.31, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=99.14%, 20=0.82%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%, '
'250=0.01%\n'
' cpu : usr=48.33%, '
'sys=51.64%, ctx=14, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1020055,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=71.2MiB/s (74.6MB/s), '
'71.2MiB/s-71.2MiB/s '
'(74.6MB/s-74.6MB/s), io=498MiB '
'(522MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_87.output'),
'state': {'begin': 1666782467.2342203,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8045120239257812,
'end': 1666782468.0387323,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_87.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_87.state')},
'cmd_88': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_88.output'),
'state': {'begin': 1666782470.2117727,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8008444309234619,
'end': 1666782471.012617,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_88.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_88.state')},
'cmd_89': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_89.output'),
'state': {'begin': 1666782471.0155253,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.44051480293274,
'end': 1666782482.4560401,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_89.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_89.state')},
'cmd_90': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789572,\n'
' "timestamp_ms" : '
'1666789572451,\n'
' "time" : "Wed Oct 26 15:06:12 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 521175040,\n'
' "io_kbytes" : 508960,\n'
' "bw_bytes" : 74442942,\n'
' "bw" : 72698,\n'
' "iops" : 145396.371947,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1017920,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1743,\n'
' "max" : 14618,\n'
' "mean" : 1896.726667,\n'
' "stddev" : 103.752460,\n'
' "N" : 1017920\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 601,\n'
' "max" : 117551,\n'
' "mean" : 4735.870113,\n'
' "stddev" : '
'1060.455946,\n'
' "N" : 1017920,\n'
' "percentile" : {\n'
' "1.000000" : 4384,\n'
' "5.000000" : 4384,\n'
' "10.000000" : 4448,\n'
' "20.000000" : 4448,\n'
' "30.000000" : 4640,\n'
' "40.000000" : 4640,\n'
' "50.000000" : 4640,\n'
' "60.000000" : 4640,\n'
' "70.000000" : 4704,\n'
' "80.000000" : 4704,\n'
' "90.000000" : 4704,\n'
' "95.000000" : 4896,\n'
' "99.000000" : 9792,\n'
' "99.500000" : 10816,\n'
' "99.900000" : 15168,\n'
' "99.950000" : 15424,\n'
' "99.990000" : 56576\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5891,\n'
' "max" : 119544,\n'
' "mean" : 6632.596780,\n'
' "stddev" : '
'1064.048858,\n'
' "N" : 1017920\n'
' },\n'
' "bw_min" : 72585,\n'
' "bw_max" : 72865,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'72711.142857,\n'
' "bw_dev" : 64.441813,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 145170,\n'
' "iops_max" : 145730,\n'
' "iops_mean" : '
'145422.285714,\n'
' "iops_stddev" : '
'128.883626,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 53.528571,\n'
' "sys_cpu" : 46.457143,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.016111,\n'
' "10" : 99.099733,\n'
' "20" : 0.863329,\n'
' "50" : 0.010000,\n'
' "100" : 0.011199,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=6458: Wed '
'Oct 26 15:06:12 2022\n'
' read: IOPS=145k, BW=71.0MiB/s '
'(74.4MB/s)(497MiB/7001msec)\n'
' slat (nsec): min=1743, '
'max=14618, avg=1896.73, '
'stdev=103.75\n'
' clat (nsec): min=601, '
'max=117551, avg=4735.87, '
'stdev=1060.46\n'
' lat (usec): min=5, max=119, '
'avg= 6.63, stdev= 1.06\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4384], 5.00th=[ '
'4384], 10.00th=[ 4448], 20.00th=[ '
'4448],\n'
' | 30.00th=[ 4640], 40.00th=[ '
'4640], 50.00th=[ 4640], 60.00th=[ '
'4640],\n'
' | 70.00th=[ 4704], 80.00th=[ '
'4704], 90.00th=[ 4704], 95.00th=[ '
'4896],\n'
' | 99.00th=[ 9792], '
'99.50th=[10816], 99.90th=[15168], '
'99.95th=[15424],\n'
' | 99.99th=[56576]\n'
' bw ( KiB/s): min=72585, '
'max=72865, per=100.00%, '
'avg=72711.14, stdev=64.44, '
'samples=14\n'
' iops : min=145170, '
'max=145730, avg=145422.29, '
'stdev=128.88, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.02%, 10=99.10%, 20=0.86%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%, '
'250=0.01%\n'
' cpu : usr=53.53%, '
'sys=46.46%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1017920,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=71.0MiB/s (74.4MB/s), '
'71.0MiB/s-71.0MiB/s '
'(74.4MB/s-74.4MB/s), io=497MiB '
'(521MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_90.output'),
'state': {'begin': 1666782482.4585795,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7996425628662109,
'end': 1666782483.258222,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_90.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_90.state')},
'cmd_91': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_libaio-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_91.output'),
'state': {'begin': 1666782484.3010526,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7843501567840576,
'end': 1666782485.0854027,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_91.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_91.state')},
'cmd_92': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_92.output'),
'state': {'begin': 1666782485.0873108,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.084620952606201,
'end': 1666782496.1719317,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_92.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_92.state')},
'cmd_93': {'output': 'libaio-xnvme: (g=0): rw=randread, '
'bs=(R) 512B-512B, (W) 512B-512B, '
'(T) 512B-512B, ioengine=xnvme, '
'iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789586,\n'
' "timestamp_ms" : '
'1666789586262,\n'
' "time" : "Wed Oct 26 15:06:26 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : "libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 384857088,\n'
' "io_kbytes" : 375837,\n'
' "bw_bytes" : 54979584,\n'
' "bw" : 53691,\n'
' "iops" : 107381.857143,\n'
' "runtime" : 7000,\n'
' "total_ios" : 751673,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1693,\n'
' "max" : 46567,\n'
' "mean" : 1810.359939,\n'
' "stddev" : 98.872126,\n'
' "N" : 751674\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 612,\n'
' "max" : 75983,\n'
' "mean" : 7221.342058,\n'
' "stddev" : 966.759816,\n'
' "N" : 751673,\n'
' "percentile" : {\n'
' "1.000000" : 6752,\n'
' "5.000000" : 6944,\n'
' "10.000000" : 7008,\n'
' "20.000000" : 7008,\n'
' "30.000000" : 7072,\n'
' "40.000000" : 7072,\n'
' "50.000000" : 7136,\n'
' "60.000000" : 7200,\n'
' "70.000000" : 7200,\n'
' "80.000000" : 7200,\n'
' "90.000000" : 7264,\n'
' "95.000000" : 7392,\n'
' "99.000000" : 10048,\n'
' "99.500000" : 13632,\n'
' "99.900000" : 15424,\n'
' "99.950000" : 15808,\n'
' "99.990000" : 57600\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7604,\n'
' "max" : 77766,\n'
' "mean" : 9031.702033,\n'
' "stddev" : 973.215408,\n'
' "N" : 751673\n'
' },\n'
' "bw_min" : 53585,\n'
' "bw_max" : 53789,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'53692.785714,\n'
' "bw_dev" : 71.730455,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 107170,\n'
' "iops_max" : 107578,\n'
' "iops_mean" : '
'107385.642857,\n'
' "iops_stddev" : '
'143.549298,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 11.500000,\n'
' "sys_cpu" : 35.457143,\n'
' "ctx" : 751676,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 98.988124,\n'
' "20" : 0.992320,\n'
' "50" : 0.010000,\n'
' "100" : 0.014767,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=6506: Wed '
'Oct 26 15:06:26 2022\n'
' read: IOPS=107k, BW=52.4MiB/s '
'(55.0MB/s)(367MiB/7000msec)\n'
' slat (nsec): min=1693, '
'max=46567, avg=1810.36, '
'stdev=98.87\n'
' clat (nsec): min=612, '
'max=75983, avg=7221.34, '
'stdev=966.76\n'
' lat (nsec): min=7604, '
'max=77766, avg=9031.70, '
'stdev=973.22\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6752], 5.00th=[ '
'6944], 10.00th=[ 7008], 20.00th=[ '
'7008],\n'
' | 30.00th=[ 7072], 40.00th=[ '
'7072], 50.00th=[ 7136], 60.00th=[ '
'7200],\n'
' | 70.00th=[ 7200], 80.00th=[ '
'7200], 90.00th=[ 7264], 95.00th=[ '
'7392],\n'
' | 99.00th=[10048], '
'99.50th=[13632], 99.90th=[15424], '
'99.95th=[15808],\n'
' | 99.99th=[57600]\n'
' bw ( KiB/s): min=53585, '
'max=53789, per=100.00%, '
'avg=53692.79, stdev=71.73, '
'samples=14\n'
' iops : min=107170, '
'max=107578, avg=107385.64, '
'stdev=143.55, samples=14\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=98.99%, 20=0.99%, 50=0.01%, '
'100=0.01%\n'
' cpu : usr=11.50%, '
'sys=35.46%, ctx=751676, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=751673,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=52.4MiB/s (55.0MB/s), '
'52.4MiB/s-52.4MiB/s '
'(55.0MB/s-55.0MB/s), io=367MiB '
'(385MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_93.output'),
'state': {'begin': 1666782496.1747887,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8476107120513916,
'end': 1666782497.0223994,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_93.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_93.state')},
'cmd_94': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_libaio-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_94.output'),
'state': {'begin': 1666782498.1266768,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8207857608795166,
'end': 1666782498.9474626,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_94.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_94.state')},
'cmd_95': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_95.output'),
'state': {'begin': 1666782498.949676,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.12022852897644,
'end': 1666782510.0699046,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_95.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_95.state')},
'cmd_96': {'output': 'libaio-xnvme: (g=0): rw=randread, '
'bs=(R) 512B-512B, (W) 512B-512B, '
'(T) 512B-512B, ioengine=xnvme, '
'iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789600,\n'
' "timestamp_ms" : '
'1666789600159,\n'
' "time" : "Wed Oct 26 15:06:40 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : "libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 385519104,\n'
' "io_kbytes" : 376483,\n'
' "bw_bytes" : 55066291,\n'
' "bw" : 53775,\n'
' "iops" : 107551.349807,\n'
' "runtime" : 7001,\n'
' "total_ios" : 752967,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1693,\n'
' "max" : 22682,\n'
' "mean" : 1809.645753,\n'
' "stddev" : 88.186978,\n'
' "N" : 752967\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 672,\n'
' "max" : 75631,\n'
' "mean" : 7207.420906,\n'
' "stddev" : '
'1021.305121,\n'
' "N" : 752967,\n'
' "percentile" : {\n'
' "1.000000" : 6752,\n'
' "5.000000" : 6944,\n'
' "10.000000" : 6944,\n'
' "20.000000" : 7008,\n'
' "30.000000" : 7008,\n'
' "40.000000" : 7072,\n'
' "50.000000" : 7136,\n'
' "60.000000" : 7136,\n'
' "70.000000" : 7200,\n'
' "80.000000" : 7200,\n'
' "90.000000" : 7264,\n'
' "95.000000" : 7392,\n'
' "99.000000" : 10048,\n'
' "99.500000" : 13504,\n'
' "99.900000" : 15424,\n'
' "99.950000" : 15808,\n'
' "99.990000" : 60160\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7604,\n'
' "max" : 77385,\n'
' "mean" : 9017.066659,\n'
' "stddev" : '
'1026.795108,\n'
' "N" : 752967\n'
' },\n'
' "bw_min" : 53671,\n'
' "bw_max" : 53904,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'53804.928571,\n'
' "bw_dev" : 70.010400,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 107342,\n'
' "iops_max" : 107808,\n'
' "iops_mean" : '
'107610.000000,\n'
' "iops_stddev" : '
'139.982966,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 11.300000,\n'
' "sys_cpu" : 35.500000,\n'
' "ctx" : 752967,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 98.972863,\n'
' "20" : 1.005755,\n'
' "50" : 0.010000,\n'
' "100" : 0.017265,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=6554: Wed '
'Oct 26 15:06:40 2022\n'
' read: IOPS=108k, BW=52.5MiB/s '
'(55.1MB/s)(368MiB/7001msec)\n'
' slat (nsec): min=1693, '
'max=22682, avg=1809.65, '
'stdev=88.19\n'
' clat (nsec): min=672, '
'max=75631, avg=7207.42, '
'stdev=1021.31\n'
' lat (nsec): min=7604, '
'max=77385, avg=9017.07, '
'stdev=1026.80\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6752], 5.00th=[ '
'6944], 10.00th=[ 6944], 20.00th=[ '
'7008],\n'
' | 30.00th=[ 7008], 40.00th=[ '
'7072], 50.00th=[ 7136], 60.00th=[ '
'7136],\n'
' | 70.00th=[ 7200], 80.00th=[ '
'7200], 90.00th=[ 7264], 95.00th=[ '
'7392],\n'
' | 99.00th=[10048], '
'99.50th=[13504], 99.90th=[15424], '
'99.95th=[15808],\n'
' | 99.99th=[60160]\n'
' bw ( KiB/s): min=53671, '
'max=53904, per=100.00%, '
'avg=53804.93, stdev=70.01, '
'samples=14\n'
' iops : min=107342, '
'max=107808, avg=107610.00, '
'stdev=139.98, samples=14\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=98.97%, 20=1.01%, 50=0.01%, '
'100=0.02%\n'
' cpu : usr=11.30%, '
'sys=35.50%, ctx=752967, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=752967,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=52.5MiB/s (55.1MB/s), '
'52.5MiB/s-52.5MiB/s '
'(55.1MB/s-55.1MB/s), io=368MiB '
'(386MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_96.output'),
'state': {'begin': 1666782510.072631,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8313148021697998,
'end': 1666782510.9039457,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_96.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_96.state')},
'cmd_97': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_libaio-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_97.output'),
'state': {'begin': 1666782512.010404,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8171346187591553,
'end': 1666782512.8275387,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_97.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_97.state')},
'cmd_98': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_98.output'),
'state': {'begin': 1666782512.8297102,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.117222547531128,
'end': 1666782523.9469328,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_98.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_98.state')},
'cmd_99': {'output': 'libaio-xnvme: (g=0): rw=randread, '
'bs=(R) 512B-512B, (W) 512B-512B, '
'(T) 512B-512B, ioengine=xnvme, '
'iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666789614,\n'
' "timestamp_ms" : '
'1666789614006,\n'
' "time" : "Wed Oct 26 15:06:54 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : "libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 382008832,\n'
' "io_kbytes" : 373055,\n'
' "bw_bytes" : 54564895,\n'
' "bw" : 53286,\n'
' "iops" : 106571.918297,\n'
' "runtime" : 7001,\n'
' "total_ios" : 746110,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1743,\n'
' "max" : 20578,\n'
' "mean" : 1853.316921,\n'
' "stddev" : 84.943754,\n'
' "N" : 746111\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 571,\n'
' "max" : 462377,\n'
' "mean" : 7237.205919,\n'
' "stddev" : '
'1091.433101,\n'
' "N" : 746110,\n'
' "percentile" : {\n'
' "1.000000" : 6816,\n'
' "5.000000" : 6944,\n'
' "10.000000" : 7008,\n'
' "20.000000" : 7072,\n'
' "30.000000" : 7072,\n'
' "40.000000" : 7072,\n'
' "50.000000" : 7136,\n'
' "60.000000" : 7200,\n'
' "70.000000" : 7200,\n'
' "80.000000" : 7264,\n'
' "90.000000" : 7328,\n'
' "95.000000" : 7456,\n'
' "99.000000" : 9920,\n'
' "99.500000" : 13632,\n'
' "99.900000" : 15424,\n'
' "99.950000" : 16064,\n'
' "99.990000" : 57600\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6503,\n'
' "max" : 464231,\n'
' "mean" : 9090.522852,\n'
' "stddev" : '
'1096.371180,\n'
' "N" : 746110\n'
' },\n'
' "bw_min" : 53220,\n'
' "bw_max" : 53390,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'53313.285714,\n'
' "bw_dev" : 55.447034,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 106440,\n'
' "iops_max" : 106780,\n'
' "iops_mean" : '
'106626.714286,\n'
' "iops_stddev" : '
'111.025589,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.171429,\n'
' "sys_cpu" : 34.214286,\n'
' "ctx" : 746110,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 99.015829,\n'
' "20" : 0.965541,\n'
' "50" : 0.010000,\n'
' "100" : 0.013403,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=6602: Wed '
'Oct 26 15:06:53 2022\n'
' read: IOPS=107k, BW=52.0MiB/s '
'(54.6MB/s)(364MiB/7001msec)\n'
' slat (nsec): min=1743, '
'max=20578, avg=1853.32, '
'stdev=84.94\n'
' clat (nsec): min=571, '
'max=462377, avg=7237.21, '
'stdev=1091.43\n'
' lat (usec): min=6, max=464, '
'avg= 9.09, stdev= 1.10\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6816], 5.00th=[ '
'6944], 10.00th=[ 7008], 20.00th=[ '
'7072],\n'
' | 30.00th=[ 7072], 40.00th=[ '
'7072], 50.00th=[ 7136], 60.00th=[ '
'7200],\n'
' | 70.00th=[ 7200], 80.00th=[ '
'7264], 90.00th=[ 7328], 95.00th=[ '
'7456],\n'
' | 99.00th=[ 9920], '
'99.50th=[13632], 99.90th=[15424], '
'99.95th=[16064],\n'
' | 99.99th=[57600]\n'
' bw ( KiB/s): min=53220, '
'max=53390, per=100.00%, '
'avg=53313.29, stdev=55.45, '
'samples=14\n'
' iops : min=106440, '
'max=106780, avg=106626.71, '
'stdev=111.03, samples=14\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=99.02%, 20=0.97%, 50=0.01%, '
'100=0.01%\n'
' lat (usec) : 500=0.01%\n'
' cpu : usr=13.17%, '
'sys=34.21%, ctx=746110, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=746110,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=52.0MiB/s (54.6MB/s), '
'52.0MiB/s-52.0MiB/s '
'(54.6MB/s-54.6MB/s), io=364MiB '
'(382MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_99.output'),
'state': {'begin': 1666782523.9498014,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8183553218841553,
'end': 1666782524.7681568,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_99.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_99.state')}}},
'id': '003_fio',
'name': 'fio',
'nr': 3,
'status': {'elapsed': 1946.3082365989685,
'failed': 0,
'passed': 1,
'skipped': 0,
'started': 0.0},
'uses': 'fioe',
'with': {'iodepths': [1, 2, 4, 8], 'iosizes': ['512'], 'repetitions': 3}}
$ rm /tmp/fio-output_512_1_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789148,
"timestamp_ms" : 1666789148400,
"time" : "Wed Oct 26 14:59:08 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 394994176,
"io_kbytes" : 385736,
"bw_bytes" : 56419679,
"bw" : 55097,
"iops" : 110194.686473,
"runtime" : 7001,
"total_ios" : 771473,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1362,
"max" : 5961,
"mean" : 1460.212779,
"stddev" : 51.214525,
"N" : 771473
},
"clat_ns" : {
"min" : 2955,
"max" : 117891,
"mean" : 7271.748888,
"stddev" : 971.362791,
"N" : 771473,
"percentile" : {
"1.000000" : 6752,
"5.000000" : 7008,
"10.000000" : 7008,
"20.000000" : 7072,
"30.000000" : 7072,
"40.000000" : 7136,
"50.000000" : 7200,
"60.000000" : 7200,
"70.000000" : 7264,
"80.000000" : 7264,
"90.000000" : 7392,
"95.000000" : 7456,
"99.000000" : 10432,
"99.500000" : 13632,
"99.900000" : 15680,
"99.950000" : 15808,
"99.990000" : 57600
}
},
"lat_ns" : {
"min" : 7354,
"max" : 119324,
"mean" : 8731.961667,
"stddev" : 973.813190,
"N" : 771473
},
"bw_min" : 54956,
"bw_max" : 55331,
"bw_agg" : 100.000000,
"bw_mean" : 55107.071429,
"bw_dev" : 106.053299,
"bw_samples" : 14,
"iops_min" : 109912,
"iops_max" : 110662,
"iops_mean" : 110214.142857,
"iops_stddev" : 212.106598,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 8.028571,
"sys_cpu" : 35.128571,
"ctx" : 771473,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 98.671243,
"20" : 1.310610,
"50" : 0.010000,
"100" : 0.012703,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=4935: Wed Oct 26 14:59:08 2022
read: IOPS=110k, BW=53.8MiB/s (56.4MB/s)(377MiB/7001msec)
slat (nsec): min=1362, max=5961, avg=1460.21, stdev=51.21
clat (nsec): min=2955, max=117891, avg=7271.75, stdev=971.36
lat (nsec): min=7354, max=119324, avg=8731.96, stdev=973.81
clat percentiles (nsec):
| 1.00th=[ 6752], 5.00th=[ 7008], 10.00th=[ 7008], 20.00th=[ 7072],
| 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7200], 60.00th=[ 7200],
| 70.00th=[ 7264], 80.00th=[ 7264], 90.00th=[ 7392], 95.00th=[ 7456],
| 99.00th=[10432], 99.50th=[13632], 99.90th=[15680], 99.95th=[15808],
| 99.99th=[57600]
bw ( KiB/s): min=54956, max=55331, per=100.00%, avg=55107.07, stdev=106.05, samples=14
iops : min=109912, max=110662, avg=110214.14, stdev=212.11, samples=14
lat (usec) : 4=0.01%, 10=98.67%, 20=1.31%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%
cpu : usr=8.03%, sys=35.13%, ctx=771473, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=771473,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=53.8MiB/s (56.4MB/s), 53.8MiB/s-53.8MiB/s (56.4MB/s-56.4MB/s), io=377MiB (395MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789162,
"timestamp_ms" : 1666789162269,
"time" : "Wed Oct 26 14:59:22 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 394686464,
"io_kbytes" : 385436,
"bw_bytes" : 56383780,
"bw" : 55062,
"iops" : 110124.571429,
"runtime" : 7000,
"total_ios" : 770872,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1402,
"max" : 13765,
"mean" : 1493.597215,
"stddev" : 57.341593,
"N" : 770872
},
"clat_ns" : {
"min" : 171,
"max" : 77646,
"mean" : 7265.220271,
"stddev" : 980.277945,
"N" : 770872,
"percentile" : {
"1.000000" : 6752,
"5.000000" : 7008,
"10.000000" : 7008,
"20.000000" : 7072,
"30.000000" : 7072,
"40.000000" : 7136,
"50.000000" : 7200,
"60.000000" : 7200,
"70.000000" : 7264,
"80.000000" : 7264,
"90.000000" : 7392,
"95.000000" : 7456,
"99.000000" : 10560,
"99.500000" : 13504,
"99.900000" : 15680,
"99.950000" : 15808,
"99.990000" : 59136
}
},
"lat_ns" : {
"min" : 7434,
"max" : 79108,
"mean" : 8758.817486,
"stddev" : 983.505773,
"N" : 770872
},
"bw_min" : 54999,
"bw_max" : 55216,
"bw_agg" : 100.000000,
"bw_mean" : 55079.642857,
"bw_dev" : 51.880939,
"bw_samples" : 14,
"iops_min" : 109998,
"iops_max" : 110432,
"iops_mean" : 110159.285714,
"iops_stddev" : 103.761879,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 6.757143,
"sys_cpu" : 36.657143,
"ctx" : 770872,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 98.658921,
"20" : 1.323436,
"50" : 0.010000,
"100" : 0.013751,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=4981: Wed Oct 26 14:59:22 2022
read: IOPS=110k, BW=53.8MiB/s (56.4MB/s)(376MiB/7000msec)
slat (nsec): min=1402, max=13765, avg=1493.60, stdev=57.34
clat (nsec): min=171, max=77646, avg=7265.22, stdev=980.28
lat (nsec): min=7434, max=79108, avg=8758.82, stdev=983.51
clat percentiles (nsec):
| 1.00th=[ 6752], 5.00th=[ 7008], 10.00th=[ 7008], 20.00th=[ 7072],
| 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7200], 60.00th=[ 7200],
| 70.00th=[ 7264], 80.00th=[ 7264], 90.00th=[ 7392], 95.00th=[ 7456],
| 99.00th=[10560], 99.50th=[13504], 99.90th=[15680], 99.95th=[15808],
| 99.99th=[59136]
bw ( KiB/s): min=54999, max=55216, per=100.00%, avg=55079.64, stdev=51.88, samples=14
iops : min=109998, max=110432, avg=110159.29, stdev=103.76, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=98.66%, 20=1.32%, 50=0.01%, 100=0.01%
cpu : usr=6.76%, sys=36.66%, ctx=770872, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=770872,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=53.8MiB/s (56.4MB/s), 53.8MiB/s-53.8MiB/s (56.4MB/s-56.4MB/s), io=376MiB (395MB), run=7000-7000msec
$ rm /tmp/fio-output_512_1_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789176,
"timestamp_ms" : 1666789176144,
"time" : "Wed Oct 26 14:59:36 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 397914112,
"io_kbytes" : 388588,
"bw_bytes" : 56836753,
"bw" : 55504,
"iops" : 111009.141551,
"runtime" : 7001,
"total_ios" : 777175,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1352,
"max" : 16331,
"mean" : 1435.588716,
"stddev" : 64.972788,
"N" : 777176
},
"clat_ns" : {
"min" : 200,
"max" : 118251,
"mean" : 7248.129245,
"stddev" : 1059.290764,
"N" : 777175,
"percentile" : {
"1.000000" : 6752,
"5.000000" : 6944,
"10.000000" : 7008,
"20.000000" : 7008,
"30.000000" : 7072,
"40.000000" : 7072,
"50.000000" : 7136,
"60.000000" : 7200,
"70.000000" : 7200,
"80.000000" : 7264,
"90.000000" : 7328,
"95.000000" : 7456,
"99.000000" : 10688,
"99.500000" : 13632,
"99.900000" : 15680,
"99.950000" : 15936,
"99.990000" : 60672
}
},
"lat_ns" : {
"min" : 7274,
"max" : 119724,
"mean" : 8683.718016,
"stddev" : 1062.030108,
"N" : 777175
},
"bw_min" : 55395,
"bw_max" : 55748,
"bw_agg" : 100.000000,
"bw_mean" : 55535.571429,
"bw_dev" : 121.563984,
"bw_samples" : 14,
"iops_min" : 110790,
"iops_max" : 111496,
"iops_mean" : 111071.285714,
"iops_stddev" : 243.041443,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 7.557143,
"sys_cpu" : 35.600000,
"ctx" : 777179,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 98.510245,
"20" : 1.465307,
"50" : 0.010000,
"100" : 0.017757,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=5027: Wed Oct 26 14:59:36 2022
read: IOPS=111k, BW=54.2MiB/s (56.8MB/s)(379MiB/7001msec)
slat (nsec): min=1352, max=16331, avg=1435.59, stdev=64.97
clat (nsec): min=200, max=118251, avg=7248.13, stdev=1059.29
lat (usec): min=7, max=119, avg= 8.68, stdev= 1.06
clat percentiles (nsec):
| 1.00th=[ 6752], 5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7008],
| 30.00th=[ 7072], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7200],
| 70.00th=[ 7200], 80.00th=[ 7264], 90.00th=[ 7328], 95.00th=[ 7456],
| 99.00th=[10688], 99.50th=[13632], 99.90th=[15680], 99.95th=[15936],
| 99.99th=[60672]
bw ( KiB/s): min=55395, max=55748, per=100.00%, avg=55535.57, stdev=121.56, samples=14
iops : min=110790, max=111496, avg=111071.29, stdev=243.04, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=98.51%, 20=1.47%, 50=0.01%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=7.56%, sys=35.60%, ctx=777179, majf=0, minf=1
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=777175,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=54.2MiB/s (56.8MB/s), 54.2MiB/s-54.2MiB/s (56.8MB/s-56.8MB/s), io=379MiB (398MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt': No such file or directory
$ rm /tmp/fio-output_512_2_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789627,
"timestamp_ms" : 1666789627760,
"time" : "Wed Oct 26 15:07:07 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 762248704,
"io_kbytes" : 744383,
"bw_bytes" : 108877118,
"bw" : 106325,
"iops" : 212650.478503,
"runtime" : 7001,
"total_ios" : 1488766,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1172,
"max" : 21300,
"mean" : 1340.914016,
"stddev" : 138.459284,
"N" : 1488766
},
"clat_ns" : {
"min" : 151,
"max" : 118081,
"mean" : 7781.440333,
"stddev" : 2292.078431,
"N" : 1488767,
"percentile" : {
"1.000000" : 5664,
"5.000000" : 5792,
"10.000000" : 5792,
"20.000000" : 5856,
"30.000000" : 5856,
"40.000000" : 5920,
"50.000000" : 9024,
"60.000000" : 9152,
"70.000000" : 9280,
"80.000000" : 9280,
"90.000000" : 9408,
"95.000000" : 9536,
"99.000000" : 16768,
"99.500000" : 18048,
"99.900000" : 19072,
"99.950000" : 23936,
"99.990000" : 64768
}
},
"lat_ns" : {
"min" : 5661,
"max" : 119274,
"mean" : 9122.354383,
"stddev" : 2373.673269,
"N" : 1488767
},
"bw_min" : 106162,
"bw_max" : 106787,
"bw_agg" : 100.000000,
"bw_mean" : 106405.307692,
"bw_dev" : 184.502206,
"bw_samples" : 13,
"iops_min" : 212324,
"iops_max" : 213575,
"iops_mean" : 212810.769231,
"iops_stddev" : 369.150185,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.371429,
"sys_cpu" : 46.300000,
"ctx" : 746929,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.396434,
"20" : 2.514230,
"50" : 0.056221,
"100" : 0.025928,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=6649: Wed Oct 26 15:07:07 2022
read: IOPS=213k, BW=104MiB/s (109MB/s)(727MiB/7001msec)
slat (nsec): min=1172, max=21300, avg=1340.91, stdev=138.46
clat (nsec): min=151, max=118081, avg=7781.44, stdev=2292.08
lat (usec): min=5, max=119, avg= 9.12, stdev= 2.37
clat percentiles (nsec):
| 1.00th=[ 5664], 5.00th=[ 5792], 10.00th=[ 5792], 20.00th=[ 5856],
| 30.00th=[ 5856], 40.00th=[ 5920], 50.00th=[ 9024], 60.00th=[ 9152],
| 70.00th=[ 9280], 80.00th=[ 9280], 90.00th=[ 9408], 95.00th=[ 9536],
| 99.00th=[16768], 99.50th=[18048], 99.90th=[19072], 99.95th=[23936],
| 99.99th=[64768]
bw ( KiB/s): min=106162, max=106787, per=100.00%, avg=106405.31, stdev=184.50, samples=13
iops : min=212324, max=213575, avg=212810.77, stdev=369.15, samples=13
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=97.40%, 20=2.51%, 50=0.06%, 100=0.03%
lat (usec) : 250=0.01%
cpu : usr=14.37%, sys=46.30%, ctx=746929, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1488766,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=727MiB (762MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789641,
"timestamp_ms" : 1666789641533,
"time" : "Wed Oct 26 15:07:21 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 763893760,
"io_kbytes" : 745990,
"bw_bytes" : 109112092,
"bw" : 106554,
"iops" : 213109.270104,
"runtime" : 7001,
"total_ios" : 1491978,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1162,
"max" : 46527,
"mean" : 1340.729218,
"stddev" : 143.970424,
"N" : 1491979
},
"clat_ns" : {
"min" : 150,
"max" : 118172,
"mean" : 7760.151302,
"stddev" : 2264.115071,
"N" : 1491979,
"percentile" : {
"1.000000" : 5600,
"5.000000" : 5792,
"10.000000" : 5792,
"20.000000" : 5856,
"30.000000" : 5856,
"40.000000" : 5920,
"50.000000" : 8896,
"60.000000" : 9152,
"70.000000" : 9280,
"80.000000" : 9280,
"90.000000" : 9408,
"95.000000" : 9536,
"99.000000" : 16768,
"99.500000" : 18048,
"99.900000" : 18816,
"99.950000" : 23680,
"99.990000" : 60672
}
},
"lat_ns" : {
"min" : 5590,
"max" : 119384,
"mean" : 9100.880520,
"stddev" : 2350.258010,
"N" : 1491979
},
"bw_min" : 106122,
"bw_max" : 106923,
"bw_agg" : 100.000000,
"bw_mean" : 106573.500000,
"bw_dev" : 240.951559,
"bw_samples" : 14,
"iops_min" : 212244,
"iops_max" : 213846,
"iops_mean" : 213147.000000,
"iops_stddev" : 481.903118,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.057143,
"sys_cpu" : 46.657143,
"ctx" : 748714,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.423286,
"20" : 2.503053,
"50" : 0.046985,
"100" : 0.020979,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=6695: Wed Oct 26 15:07:21 2022
read: IOPS=213k, BW=104MiB/s (109MB/s)(729MiB/7001msec)
slat (nsec): min=1162, max=46527, avg=1340.73, stdev=143.97
clat (nsec): min=150, max=118172, avg=7760.15, stdev=2264.12
lat (usec): min=5, max=119, avg= 9.10, stdev= 2.35
clat percentiles (nsec):
| 1.00th=[ 5600], 5.00th=[ 5792], 10.00th=[ 5792], 20.00th=[ 5856],
| 30.00th=[ 5856], 40.00th=[ 5920], 50.00th=[ 8896], 60.00th=[ 9152],
| 70.00th=[ 9280], 80.00th=[ 9280], 90.00th=[ 9408], 95.00th=[ 9536],
| 99.00th=[16768], 99.50th=[18048], 99.90th=[18816], 99.95th=[23680],
| 99.99th=[60672]
bw ( KiB/s): min=106122, max=106923, per=100.00%, avg=106573.50, stdev=240.95, samples=14
iops : min=212244, max=213846, avg=213147.00, stdev=481.90, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=97.42%, 20=2.50%, 50=0.05%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=14.06%, sys=46.66%, ctx=748714, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1491978,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=729MiB (764MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789655,
"timestamp_ms" : 1666789655332,
"time" : "Wed Oct 26 15:07:35 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 764250112,
"io_kbytes" : 746338,
"bw_bytes" : 109162992,
"bw" : 106604,
"iops" : 213208.827310,
"runtime" : 7001,
"total_ios" : 1492675,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1162,
"max" : 25678,
"mean" : 1346.666306,
"stddev" : 139.012440,
"N" : 1492675
},
"clat_ns" : {
"min" : 90,
"max" : 244338,
"mean" : 7752.355304,
"stddev" : 2295.470051,
"N" : 1492676,
"percentile" : {
"1.000000" : 5600,
"5.000000" : 5728,
"10.000000" : 5792,
"20.000000" : 5792,
"30.000000" : 5856,
"40.000000" : 5920,
"50.000000" : 9024,
"60.000000" : 9152,
"70.000000" : 9280,
"80.000000" : 9280,
"90.000000" : 9408,
"95.000000" : 9536,
"99.000000" : 16768,
"99.500000" : 18048,
"99.900000" : 18816,
"99.950000" : 23680,
"99.990000" : 63232
}
},
"lat_ns" : {
"min" : 5621,
"max" : 245951,
"mean" : 9099.021662,
"stddev" : 2381.626970,
"N" : 1492676
},
"bw_min" : 106186,
"bw_max" : 107335,
"bw_agg" : 100.000000,
"bw_mean" : 106623.785714,
"bw_dev" : 356.455883,
"bw_samples" : 14,
"iops_min" : 212372,
"iops_max" : 214671,
"iops_mean" : 213247.642857,
"iops_stddev" : 713.065280,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 12.200000,
"sys_cpu" : 48.442857,
"ctx" : 748942,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 97.413905,
"20" : 2.505435,
"50" : 0.049776,
"100" : 0.022912,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=6741: Wed Oct 26 15:07:35 2022
read: IOPS=213k, BW=104MiB/s (109MB/s)(729MiB/7001msec)
slat (nsec): min=1162, max=25678, avg=1346.67, stdev=139.01
clat (nsec): min=90, max=244338, avg=7752.36, stdev=2295.47
lat (usec): min=5, max=245, avg= 9.10, stdev= 2.38
clat percentiles (nsec):
| 1.00th=[ 5600], 5.00th=[ 5728], 10.00th=[ 5792], 20.00th=[ 5792],
| 30.00th=[ 5856], 40.00th=[ 5920], 50.00th=[ 9024], 60.00th=[ 9152],
| 70.00th=[ 9280], 80.00th=[ 9280], 90.00th=[ 9408], 95.00th=[ 9536],
| 99.00th=[16768], 99.50th=[18048], 99.90th=[18816], 99.95th=[23680],
| 99.99th=[63232]
bw ( KiB/s): min=106186, max=107335, per=100.00%, avg=106623.79, stdev=356.46, samples=14
iops : min=212372, max=214671, avg=213247.64, stdev=713.07, samples=14
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=97.41%, 20=2.51%, 50=0.05%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=12.20%, sys=48.44%, ctx=748942, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1492675,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=729MiB (764MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789669,
"timestamp_ms" : 1666789669194,
"time" : "Wed Oct 26 15:07:49 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 760012288,
"io_kbytes" : 742199,
"bw_bytes" : 108557675,
"bw" : 106013,
"iops" : 212026.424796,
"runtime" : 7001,
"total_ios" : 1484397,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1222,
"max" : 22522,
"mean" : 1423.243689,
"stddev" : 173.920884,
"N" : 1484398
},
"clat_ns" : {
"min" : 151,
"max" : 115626,
"mean" : 7748.459844,
"stddev" : 2130.869047,
"N" : 1484398,
"percentile" : {
"1.000000" : 5536,
"5.000000" : 5664,
"10.000000" : 5728,
"20.000000" : 5792,
"30.000000" : 6176,
"40.000000" : 7328,
"50.000000" : 7520,
"60.000000" : 7712,
"70.000000" : 9280,
"80.000000" : 9408,
"90.000000" : 9536,
"95.000000" : 9664,
"99.000000" : 16768,
"99.500000" : 18304,
"99.900000" : 23936,
"99.950000" : 24448,
"99.990000" : 61184
}
},
"lat_ns" : {
"min" : 5751,
"max" : 116909,
"mean" : 9171.703695,
"stddev" : 2188.738867,
"N" : 1484398
},
"bw_min" : 105599,
"bw_max" : 106438,
"bw_agg" : 100.000000,
"bw_mean" : 106032.142857,
"bw_dev" : 187.439367,
"bw_samples" : 14,
"iops_min" : 211198,
"iops_max" : 212876,
"iops_mean" : 212064.285714,
"iops_stddev" : 374.878735,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.985714,
"sys_cpu" : 46.657143,
"ctx" : 745370,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.303619,
"20" : 2.542110,
"50" : 0.122811,
"100" : 0.021288,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=6787: Wed Oct 26 15:07:49 2022
read: IOPS=212k, BW=104MiB/s (109MB/s)(725MiB/7001msec)
slat (nsec): min=1222, max=22522, avg=1423.24, stdev=173.92
clat (nsec): min=151, max=115626, avg=7748.46, stdev=2130.87
lat (usec): min=5, max=116, avg= 9.17, stdev= 2.19
clat percentiles (nsec):
| 1.00th=[ 5536], 5.00th=[ 5664], 10.00th=[ 5728], 20.00th=[ 5792],
| 30.00th=[ 6176], 40.00th=[ 7328], 50.00th=[ 7520], 60.00th=[ 7712],
| 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[ 9536], 95.00th=[ 9664],
| 99.00th=[16768], 99.50th=[18304], 99.90th=[23936], 99.95th=[24448],
| 99.99th=[61184]
bw ( KiB/s): min=105599, max=106438, per=100.00%, avg=106032.14, stdev=187.44, samples=14
iops : min=211198, max=212876, avg=212064.29, stdev=374.88, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=97.30%, 20=2.54%, 50=0.12%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=15.99%, sys=46.66%, ctx=745370, majf=0, minf=1
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1484397,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=725MiB (760MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789683,
"timestamp_ms" : 1666789683112,
"time" : "Wed Oct 26 15:08:03 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 762302976,
"io_kbytes" : 744436,
"bw_bytes" : 108884870,
"bw" : 106332,
"iops" : 212665.619197,
"runtime" : 7001,
"total_ios" : 1488872,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1222,
"max" : 13836,
"mean" : 1386.090252,
"stddev" : 141.744320,
"N" : 1488872
},
"clat_ns" : {
"min" : 1954,
"max" : 79479,
"mean" : 7760.864534,
"stddev" : 2225.390339,
"N" : 1488873,
"percentile" : {
"1.000000" : 5536,
"5.000000" : 5728,
"10.000000" : 5728,
"20.000000" : 5792,
"30.000000" : 5856,
"40.000000" : 6496,
"50.000000" : 7520,
"60.000000" : 9152,
"70.000000" : 9280,
"80.000000" : 9408,
"90.000000" : 9408,
"95.000000" : 9536,
"99.000000" : 16768,
"99.500000" : 18304,
"99.900000" : 22400,
"99.950000" : 24192,
"99.990000" : 61696
}
},
"lat_ns" : {
"min" : 5721,
"max" : 80751,
"mean" : 9146.954825,
"stddev" : 2289.506518,
"N" : 1488873
},
"bw_min" : 106032,
"bw_max" : 106788,
"bw_agg" : 100.000000,
"bw_mean" : 106351.571429,
"bw_dev" : 183.468176,
"bw_samples" : 14,
"iops_min" : 212064,
"iops_max" : 213577,
"iops_mean" : 212703.214286,
"iops_stddev" : 367.119385,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.785714,
"sys_cpu" : 46.285714,
"ctx" : 747296,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 97.373313,
"20" : 2.508879,
"50" : 0.086441,
"100" : 0.022433,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=6834: Wed Oct 26 15:08:03 2022
read: IOPS=213k, BW=104MiB/s (109MB/s)(727MiB/7001msec)
slat (nsec): min=1222, max=13836, avg=1386.09, stdev=141.74
clat (nsec): min=1954, max=79479, avg=7760.86, stdev=2225.39
lat (nsec): min=5721, max=80751, avg=9146.95, stdev=2289.51
clat percentiles (nsec):
| 1.00th=[ 5536], 5.00th=[ 5728], 10.00th=[ 5728], 20.00th=[ 5792],
| 30.00th=[ 5856], 40.00th=[ 6496], 50.00th=[ 7520], 60.00th=[ 9152],
| 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[ 9408], 95.00th=[ 9536],
| 99.00th=[16768], 99.50th=[18304], 99.90th=[22400], 99.95th=[24192],
| 99.99th=[61696]
bw ( KiB/s): min=106032, max=106788, per=100.00%, avg=106351.57, stdev=183.47, samples=14
iops : min=212064, max=213577, avg=212703.21, stdev=367.12, samples=14
lat (usec) : 2=0.01%, 4=0.01%, 10=97.37%, 20=2.51%, 50=0.09%
lat (usec) : 100=0.02%
cpu : usr=15.79%, sys=46.29%, ctx=747296, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1488872,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=727MiB (762MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789696,
"timestamp_ms" : 1666789696879,
"time" : "Wed Oct 26 15:08:16 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 762843136,
"io_kbytes" : 744964,
"bw_bytes" : 108977590,
"bw" : 106423,
"iops" : 212846.714286,
"runtime" : 7000,
"total_ios" : 1489927,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1212,
"max" : 34444,
"mean" : 1402.238872,
"stddev" : 161.624220,
"N" : 1489927
},
"clat_ns" : {
"min" : 180,
"max" : 110517,
"mean" : 7730.380643,
"stddev" : 2149.553208,
"N" : 1489928,
"percentile" : {
"1.000000" : 5536,
"5.000000" : 5664,
"10.000000" : 5728,
"20.000000" : 5792,
"30.000000" : 5920,
"40.000000" : 7264,
"50.000000" : 7520,
"60.000000" : 7712,
"70.000000" : 9280,
"80.000000" : 9408,
"90.000000" : 9408,
"95.000000" : 9536,
"99.000000" : 16768,
"99.500000" : 18304,
"99.900000" : 23424,
"99.950000" : 24192,
"99.990000" : 63744
}
},
"lat_ns" : {
"min" : 5690,
"max" : 111760,
"mean" : 9132.619582,
"stddev" : 2205.875773,
"N" : 1489928
},
"bw_min" : 106063,
"bw_max" : 106683,
"bw_agg" : 100.000000,
"bw_mean" : 106427.071429,
"bw_dev" : 168.812807,
"bw_samples" : 14,
"iops_min" : 212126,
"iops_max" : 213366,
"iops_mean" : 212854.142857,
"iops_stddev" : 337.625614,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.314286,
"sys_cpu" : 48.085714,
"ctx" : 747909,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.368864,
"20" : 2.503680,
"50" : 0.096649,
"100" : 0.021276,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=6880: Wed Oct 26 15:08:16 2022
read: IOPS=213k, BW=104MiB/s (109MB/s)(728MiB/7000msec)
slat (nsec): min=1212, max=34444, avg=1402.24, stdev=161.62
clat (nsec): min=180, max=110517, avg=7730.38, stdev=2149.55
lat (usec): min=5, max=111, avg= 9.13, stdev= 2.21
clat percentiles (nsec):
| 1.00th=[ 5536], 5.00th=[ 5664], 10.00th=[ 5728], 20.00th=[ 5792],
| 30.00th=[ 5920], 40.00th=[ 7264], 50.00th=[ 7520], 60.00th=[ 7712],
| 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[ 9408], 95.00th=[ 9536],
| 99.00th=[16768], 99.50th=[18304], 99.90th=[23424], 99.95th=[24192],
| 99.99th=[63744]
bw ( KiB/s): min=106063, max=106683, per=100.00%, avg=106427.07, stdev=168.81, samples=14
iops : min=212126, max=213366, avg=212854.14, stdev=337.63, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=97.37%, 20=2.50%, 50=0.10%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=14.31%, sys=48.09%, ctx=747909, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1489927,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=728MiB (763MB), run=7000-7000msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789189,
"timestamp_ms" : 1666789189890,
"time" : "Wed Oct 26 14:59:49 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 394787328,
"io_kbytes" : 385534,
"bw_bytes" : 56390133,
"bw" : 55068,
"iops" : 110136.980431,
"runtime" : 7001,
"total_ios" : 771069,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1402,
"max" : 10500,
"mean" : 1496.382610,
"stddev" : 66.312543,
"N" : 771069
},
"clat_ns" : {
"min" : 211,
"max" : 101751,
"mean" : 7283.762805,
"stddev" : 1020.270135,
"N" : 771069,
"percentile" : {
"1.000000" : 6752,
"5.000000" : 7008,
"10.000000" : 7072,
"20.000000" : 7072,
"30.000000" : 7072,
"40.000000" : 7136,
"50.000000" : 7200,
"60.000000" : 7200,
"70.000000" : 7264,
"80.000000" : 7264,
"90.000000" : 7392,
"95.000000" : 7456,
"99.000000" : 10560,
"99.500000" : 13504,
"99.900000" : 15680,
"99.950000" : 15936,
"99.990000" : 58624
}
},
"lat_ns" : {
"min" : 7324,
"max" : 103243,
"mean" : 8780.145415,
"stddev" : 1023.065350,
"N" : 771069
},
"bw_min" : 54847,
"bw_max" : 55341,
"bw_agg" : 100.000000,
"bw_mean" : 55095.571429,
"bw_dev" : 155.343646,
"bw_samples" : 14,
"iops_min" : 109694,
"iops_max" : 110683,
"iops_mean" : 110191.142857,
"iops_stddev" : 310.701404,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 10.771429,
"sys_cpu" : 33.328571,
"ctx" : 771067,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 98.642275,
"20" : 1.335289,
"50" : 0.010000,
"100" : 0.016211,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=5073: Wed Oct 26 14:59:49 2022
read: IOPS=110k, BW=53.8MiB/s (56.4MB/s)(376MiB/7001msec)
slat (nsec): min=1402, max=10500, avg=1496.38, stdev=66.31
clat (nsec): min=211, max=101751, avg=7283.76, stdev=1020.27
lat (usec): min=7, max=103, avg= 8.78, stdev= 1.02
clat percentiles (nsec):
| 1.00th=[ 6752], 5.00th=[ 7008], 10.00th=[ 7072], 20.00th=[ 7072],
| 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7200], 60.00th=[ 7200],
| 70.00th=[ 7264], 80.00th=[ 7264], 90.00th=[ 7392], 95.00th=[ 7456],
| 99.00th=[10560], 99.50th=[13504], 99.90th=[15680], 99.95th=[15936],
| 99.99th=[58624]
bw ( KiB/s): min=54847, max=55341, per=100.00%, avg=55095.57, stdev=155.34, samples=14
iops : min=109694, max=110683, avg=110191.14, stdev=310.70, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=98.64%, 20=1.34%, 50=0.01%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=10.77%, sys=33.33%, ctx=771067, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=771069,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=53.8MiB/s (56.4MB/s), 53.8MiB/s-53.8MiB/s (56.4MB/s-56.4MB/s), io=376MiB (395MB), run=7001-7001msec
$ cat /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789712,
"timestamp_ms" : 1666789712219,
"time" : "Wed Oct 26 15:08:32 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 909902848,
"io_kbytes" : 888577,
"bw_bytes" : 129967554,
"bw" : 126921,
"iops" : 253842.736752,
"runtime" : 7001,
"total_ios" : 1777153,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 220,
"max" : 16330,
"mean" : 271.297074,
"stddev" : 24.250548,
"N" : 1777153
},
"clat_ns" : {
"min" : 281,
"max" : 121247,
"mean" : 7381.103358,
"stddev" : 1757.802881,
"N" : 1777154,
"percentile" : {
"1.000000" : 5984,
"5.000000" : 6240,
"10.000000" : 6368,
"20.000000" : 6624,
"30.000000" : 7072,
"40.000000" : 7136,
"50.000000" : 7200,
"60.000000" : 7264,
"70.000000" : 7328,
"80.000000" : 7520,
"90.000000" : 8032,
"95.000000" : 8256,
"99.000000" : 16064,
"99.500000" : 20352,
"99.900000" : 22144,
"99.950000" : 22400,
"99.990000" : 63232
}
},
"lat_ns" : {
"min" : 5480,
"max" : 121528,
"mean" : 7652.400431,
"stddev" : 1758.138025,
"N" : 1777154
},
"bw_min" : 125453,
"bw_max" : 129526,
"bw_agg" : 100.000000,
"bw_mean" : 126965.500000,
"bw_dev" : 1491.929507,
"bw_samples" : 14,
"iops_min" : 250906,
"iops_max" : 259052,
"iops_mean" : 253931.000000,
"iops_stddev" : 2984.065734,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 17,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 99.985314,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.055459,
"20" : 2.406096,
"50" : 0.517007,
"100" : 0.020651,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=6940: Wed Oct 26 15:08:32 2022
read: IOPS=254k, BW=124MiB/s (130MB/s)(868MiB/7001msec)
slat (nsec): min=220, max=16330, avg=271.30, stdev=24.25
clat (nsec): min=281, max=121247, avg=7381.10, stdev=1757.80
lat (usec): min=5, max=121, avg= 7.65, stdev= 1.76
clat percentiles (nsec):
| 1.00th=[ 5984], 5.00th=[ 6240], 10.00th=[ 6368], 20.00th=[ 6624],
| 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7200], 60.00th=[ 7264],
| 70.00th=[ 7328], 80.00th=[ 7520], 90.00th=[ 8032], 95.00th=[ 8256],
| 99.00th=[16064], 99.50th=[20352], 99.90th=[22144], 99.95th=[22400],
| 99.99th=[63232]
bw ( KiB/s): min=125453, max=129526, per=100.00%, avg=126965.50, stdev=1491.93, samples=14
iops : min=250906, max=259052, avg=253931.00, stdev=2984.07, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=97.06%, 20=2.41%, 50=0.52%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=17, majf=0, minf=0
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1777153,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=868MiB (910MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789727,
"timestamp_ms" : 1666789727362,
"time" : "Wed Oct 26 15:08:47 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 699008512,
"io_kbytes" : 682625,
"bw_bytes" : 99844095,
"bw" : 97503,
"iops" : 195007.856021,
"runtime" : 7001,
"total_ios" : 1365250,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 711,
"max" : 12062,
"mean" : 936.282706,
"stddev" : 69.667152,
"N" : 1365250
},
"clat_ns" : {
"min" : 5841,
"max" : 82906,
"mean" : 9006.613612,
"stddev" : 1552.777792,
"N" : 1365251,
"percentile" : {
"1.000000" : 7264,
"5.000000" : 8032,
"10.000000" : 8512,
"20.000000" : 8640,
"30.000000" : 8768,
"40.000000" : 8768,
"50.000000" : 8768,
"60.000000" : 8896,
"70.000000" : 9024,
"80.000000" : 9024,
"90.000000" : 9280,
"95.000000" : 9536,
"99.000000" : 17024,
"99.500000" : 18304,
"99.900000" : 24704,
"99.950000" : 25728,
"99.990000" : 62208
}
},
"lat_ns" : {
"min" : 7314,
"max" : 83837,
"mean" : 9942.896300,
"stddev" : 1554.645663,
"N" : 1365251
},
"bw_min" : 97305,
"bw_max" : 98105,
"bw_agg" : 100.000000,
"bw_mean" : 97526.428571,
"bw_dev" : 203.481889,
"bw_samples" : 14,
"iops_min" : 194610,
"iops_max" : 196210,
"iops_mean" : 195052.857143,
"iops_stddev" : 406.963778,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 16,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.534627,
"2" : 99.465373,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.686321,
"20" : 2.956968,
"50" : 0.336129,
"100" : 0.020656,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=6998: Wed Oct 26 15:08:47 2022
read: IOPS=195k, BW=95.2MiB/s (99.8MB/s)(667MiB/7001msec)
slat (nsec): min=711, max=12062, avg=936.28, stdev=69.67
clat (nsec): min=5841, max=82906, avg=9006.61, stdev=1552.78
lat (nsec): min=7314, max=83837, avg=9942.90, stdev=1554.65
clat percentiles (nsec):
| 1.00th=[ 7264], 5.00th=[ 8032], 10.00th=[ 8512], 20.00th=[ 8640],
| 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8768], 60.00th=[ 8896],
| 70.00th=[ 9024], 80.00th=[ 9024], 90.00th=[ 9280], 95.00th=[ 9536],
| 99.00th=[17024], 99.50th=[18304], 99.90th=[24704], 99.95th=[25728],
| 99.99th=[62208]
bw ( KiB/s): min=97305, max=98105, per=100.00%, avg=97526.43, stdev=203.48, samples=14
iops : min=194610, max=196210, avg=195052.86, stdev=406.96, samples=14
lat (usec) : 10=96.69%, 20=2.96%, 50=0.34%, 100=0.02%
cpu : usr=99.99%, sys=0.00%, ctx=16, majf=0, minf=0
IO depths : 1=0.5%, 2=99.5%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1365250,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=95.2MiB/s (99.8MB/s), 95.2MiB/s-95.2MiB/s (99.8MB/s-99.8MB/s), io=667MiB (699MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789742,
"timestamp_ms" : 1666789742679,
"time" : "Wed Oct 26 15:09:02 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 808450048,
"io_kbytes" : 789502,
"bw_bytes" : 115476367,
"bw" : 112769,
"iops" : 225539.637195,
"runtime" : 7001,
"total_ios" : 1579003,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 672,
"max" : 21169,
"mean" : 953.871964,
"stddev" : 122.852261,
"N" : 1579003
},
"clat_ns" : {
"min" : 752,
"max" : 110307,
"mean" : 7658.542619,
"stddev" : 1855.562506,
"N" : 1579004,
"percentile" : {
"1.000000" : 6624,
"5.000000" : 6752,
"10.000000" : 6816,
"20.000000" : 6944,
"30.000000" : 7072,
"40.000000" : 7136,
"50.000000" : 7200,
"60.000000" : 7264,
"70.000000" : 7392,
"80.000000" : 7520,
"90.000000" : 9536,
"95.000000" : 9792,
"99.000000" : 17280,
"99.500000" : 19328,
"99.900000" : 20352,
"99.950000" : 22912,
"99.990000" : 62208
}
},
"lat_ns" : {
"min" : 7073,
"max" : 111219,
"mean" : 8612.414576,
"stddev" : 1850.718560,
"N" : 1579004
},
"bw_min" : 112339,
"bw_max" : 113330,
"bw_agg" : 100.000000,
"bw_mean" : 112795.214286,
"bw_dev" : 372.083361,
"bw_samples" : 14,
"iops_min" : 224678,
"iops_max" : 226660,
"iops_mean" : 225590.428571,
"iops_stddev" : 744.166722,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 12,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.296453,
"2" : 99.703547,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.579234,
"20" : 3.241919,
"50" : 0.156808,
"100" : 0.021849,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=7056: Wed Oct 26 15:09:02 2022
read: IOPS=226k, BW=110MiB/s (115MB/s)(771MiB/7001msec)
slat (nsec): min=672, max=21169, avg=953.87, stdev=122.85
clat (nsec): min=752, max=110307, avg=7658.54, stdev=1855.56
lat (usec): min=7, max=111, avg= 8.61, stdev= 1.85
clat percentiles (nsec):
| 1.00th=[ 6624], 5.00th=[ 6752], 10.00th=[ 6816], 20.00th=[ 6944],
| 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7200], 60.00th=[ 7264],
| 70.00th=[ 7392], 80.00th=[ 7520], 90.00th=[ 9536], 95.00th=[ 9792],
| 99.00th=[17280], 99.50th=[19328], 99.90th=[20352], 99.95th=[22912],
| 99.99th=[62208]
bw ( KiB/s): min=112339, max=113330, per=100.00%, avg=112795.21, stdev=372.08, samples=14
iops : min=224678, max=226660, avg=225590.43, stdev=744.17, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 4=0.01%, 10=96.58%, 20=3.24%, 50=0.16%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=12, majf=0, minf=0
IO depths : 1=0.3%, 2=99.7%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1579003,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=110MiB/s (115MB/s), 110MiB/s-110MiB/s (115MB/s-115MB/s), io=771MiB (808MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789756,
"timestamp_ms" : 1666789756530,
"time" : "Wed Oct 26 15:09:16 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 763407360,
"io_kbytes" : 745515,
"bw_bytes" : 109058194,
"bw" : 106502,
"iops" : 213004.142857,
"runtime" : 7000,
"total_ios" : 1491029,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1312,
"max" : 15359,
"mean" : 1507.511643,
"stddev" : 157.070001,
"N" : 1491029
},
"clat_ns" : {
"min" : 1262,
"max" : 111940,
"mean" : 7604.344876,
"stddev" : 2256.544363,
"N" : 1491030,
"percentile" : {
"1.000000" : 5280,
"5.000000" : 5344,
"10.000000" : 5408,
"20.000000" : 5472,
"30.000000" : 5600,
"40.000000" : 7264,
"50.000000" : 7648,
"60.000000" : 8032,
"70.000000" : 9024,
"80.000000" : 9152,
"90.000000" : 9280,
"95.000000" : 9408,
"99.000000" : 16512,
"99.500000" : 18560,
"99.900000" : 23936,
"99.950000" : 24192,
"99.990000" : 62720
}
},
"lat_ns" : {
"min" : 5671,
"max" : 113352,
"mean" : 9111.856576,
"stddev" : 2313.110543,
"N" : 1491030
},
"bw_min" : 106106,
"bw_max" : 106959,
"bw_agg" : 100.000000,
"bw_mean" : 106515.714286,
"bw_dev" : 259.918457,
"bw_samples" : 14,
"iops_min" : 212212,
"iops_max" : 213918,
"iops_mean" : 213031.500000,
"iops_stddev" : 519.943007,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 12.800000,
"sys_cpu" : 50.100000,
"ctx" : 749016,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 97.365913,
"20" : 2.445761,
"50" : 0.154725,
"100" : 0.024144,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2140162,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 10972,
"write_ticks" : 0,
"in_queue" : 10972,
"util" : 99.069123
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=7103: Wed Oct 26 15:09:16 2022
read: IOPS=213k, BW=104MiB/s (109MB/s)(728MiB/7000msec)
slat (nsec): min=1312, max=15359, avg=1507.51, stdev=157.07
clat (nsec): min=1262, max=111940, avg=7604.34, stdev=2256.54
lat (usec): min=5, max=113, avg= 9.11, stdev= 2.31
clat percentiles (nsec):
| 1.00th=[ 5280], 5.00th=[ 5344], 10.00th=[ 5408], 20.00th=[ 5472],
| 30.00th=[ 5600], 40.00th=[ 7264], 50.00th=[ 7648], 60.00th=[ 8032],
| 70.00th=[ 9024], 80.00th=[ 9152], 90.00th=[ 9280], 95.00th=[ 9408],
| 99.00th=[16512], 99.50th=[18560], 99.90th=[23936], 99.95th=[24192],
| 99.99th=[62720]
bw ( KiB/s): min=106106, max=106959, per=100.00%, avg=106515.71, stdev=259.92, samples=14
iops : min=212212, max=213918, avg=213031.50, stdev=519.94, samples=14
lat (usec) : 2=0.01%, 4=0.01%, 10=97.37%, 20=2.45%, 50=0.15%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=12.80%, sys=50.10%, ctx=749016, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1491029,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=728MiB (763MB), run=7000-7000msec
Disk stats (read/write):
nvme0n1: ios=2140162/0, merge=0/0, ticks=10972/0, in_queue=10972, util=99.07%
$ rm /tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt': No such file or directory
$ rm /tmp/fio-output_512_2_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789770,
"timestamp_ms" : 1666789770326,
"time" : "Wed Oct 26 15:09:30 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 757835776,
"io_kbytes" : 740074,
"bw_bytes" : 108246789,
"bw" : 105709,
"iops" : 211419.368662,
"runtime" : 7001,
"total_ios" : 1480147,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1332,
"max" : 26660,
"mean" : 1516.955765,
"stddev" : 165.330542,
"N" : 1480147
},
"clat_ns" : {
"min" : 200,
"max" : 109896,
"mean" : 7661.495982,
"stddev" : 2256.062018,
"N" : 1480148,
"percentile" : {
"1.000000" : 5280,
"5.000000" : 5408,
"10.000000" : 5472,
"20.000000" : 5536,
"30.000000" : 5792,
"40.000000" : 7328,
"50.000000" : 7648,
"60.000000" : 7968,
"70.000000" : 9152,
"80.000000" : 9152,
"90.000000" : 9280,
"95.000000" : 9536,
"99.000000" : 16768,
"99.500000" : 18560,
"99.900000" : 23936,
"99.950000" : 24448,
"99.990000" : 65280
}
},
"lat_ns" : {
"min" : 5771,
"max" : 111358,
"mean" : 9178.451778,
"stddev" : 2304.102186,
"N" : 1480148
},
"bw_min" : 105505,
"bw_max" : 106062,
"bw_agg" : 100.000000,
"bw_mean" : 105778.000000,
"bw_dev" : 128.192764,
"bw_samples" : 14,
"iops_min" : 211010,
"iops_max" : 212124,
"iops_mean" : 211556.142857,
"iops_stddev" : 256.392385,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.585714,
"sys_cpu" : 49.757143,
"ctx" : 743789,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.308105,
"20" : 2.463809,
"50" : 0.192211,
"100" : 0.026146,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2124569,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 10947,
"write_ticks" : 0,
"in_queue" : 10947,
"util" : 99.068754
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=7150: Wed Oct 26 15:09:30 2022
read: IOPS=211k, BW=103MiB/s (108MB/s)(723MiB/7001msec)
slat (nsec): min=1332, max=26660, avg=1516.96, stdev=165.33
clat (nsec): min=200, max=109896, avg=7661.50, stdev=2256.06
lat (usec): min=5, max=111, avg= 9.18, stdev= 2.30
clat percentiles (nsec):
| 1.00th=[ 5280], 5.00th=[ 5408], 10.00th=[ 5472], 20.00th=[ 5536],
| 30.00th=[ 5792], 40.00th=[ 7328], 50.00th=[ 7648], 60.00th=[ 7968],
| 70.00th=[ 9152], 80.00th=[ 9152], 90.00th=[ 9280], 95.00th=[ 9536],
| 99.00th=[16768], 99.50th=[18560], 99.90th=[23936], 99.95th=[24448],
| 99.99th=[65280]
bw ( KiB/s): min=105505, max=106062, per=100.00%, avg=105778.00, stdev=128.19, samples=14
iops : min=211010, max=212124, avg=211556.14, stdev=256.39, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=97.31%, 20=2.46%, 50=0.19%, 100=0.03%
lat (usec) : 250=0.01%
cpu : usr=13.59%, sys=49.76%, ctx=743789, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1480147,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=723MiB (758MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2124569/0, merge=0/0, ticks=10947/0, in_queue=10947, util=99.07%
$ rm /tmp/fio-output_512_2_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789784,
"timestamp_ms" : 1666789784103,
"time" : "Wed Oct 26 15:09:44 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 760478720,
"io_kbytes" : 742655,
"bw_bytes" : 108624299,
"bw" : 106078,
"iops" : 212156.691901,
"runtime" : 7001,
"total_ios" : 1485309,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1332,
"max" : 22673,
"mean" : 1514.181856,
"stddev" : 153.990339,
"N" : 1485309
},
"clat_ns" : {
"min" : 100,
"max" : 117761,
"mean" : 7616.194574,
"stddev" : 2211.320112,
"N" : 1485310,
"percentile" : {
"1.000000" : 5280,
"5.000000" : 5344,
"10.000000" : 5344,
"20.000000" : 5472,
"30.000000" : 5664,
"40.000000" : 7328,
"50.000000" : 7648,
"60.000000" : 7968,
"70.000000" : 9024,
"80.000000" : 9152,
"90.000000" : 9280,
"95.000000" : 9408,
"99.000000" : 16512,
"99.500000" : 18560,
"99.900000" : 23680,
"99.950000" : 24192,
"99.990000" : 59648
}
},
"lat_ns" : {
"min" : 5700,
"max" : 119204,
"mean" : 9130.376490,
"stddev" : 2257.397056,
"N" : 1485310
},
"bw_min" : 105355,
"bw_max" : 106441,
"bw_agg" : 100.000000,
"bw_mean" : 106110.214286,
"bw_dev" : 316.062598,
"bw_samples" : 14,
"iops_min" : 210711,
"iops_max" : 212882,
"iops_mean" : 212220.571429,
"iops_stddev" : 631.966980,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.228571,
"sys_cpu" : 50.228571,
"ctx" : 746364,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.327155,
"20" : 2.466490,
"50" : 0.177135,
"100" : 0.019592,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2126963,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 10923,
"write_ticks" : 0,
"in_queue" : 10923,
"util" : 99.078934
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=7196: Wed Oct 26 15:09:44 2022
read: IOPS=212k, BW=104MiB/s (109MB/s)(725MiB/7001msec)
slat (nsec): min=1332, max=22673, avg=1514.18, stdev=153.99
clat (nsec): min=100, max=117761, avg=7616.19, stdev=2211.32
lat (usec): min=5, max=119, avg= 9.13, stdev= 2.26
clat percentiles (nsec):
| 1.00th=[ 5280], 5.00th=[ 5344], 10.00th=[ 5344], 20.00th=[ 5472],
| 30.00th=[ 5664], 40.00th=[ 7328], 50.00th=[ 7648], 60.00th=[ 7968],
| 70.00th=[ 9024], 80.00th=[ 9152], 90.00th=[ 9280], 95.00th=[ 9408],
| 99.00th=[16512], 99.50th=[18560], 99.90th=[23680], 99.95th=[24192],
| 99.99th=[59648]
bw ( KiB/s): min=105355, max=106441, per=100.00%, avg=106110.21, stdev=316.06, samples=14
iops : min=210711, max=212882, avg=212220.57, stdev=631.97, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=97.33%, 20=2.47%, 50=0.18%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=13.23%, sys=50.23%, ctx=746364, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1485309,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=725MiB (760MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2126963/0, merge=0/0, ticks=10923/0, in_queue=10923, util=99.08%
$ rm /tmp/fio-output_512_2_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789799,
"timestamp_ms" : 1666789799256,
"time" : "Wed Oct 26 15:09:59 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 928209408,
"io_kbytes" : 906454,
"bw_bytes" : 132582403,
"bw" : 129475,
"iops" : 258949.864305,
"runtime" : 7001,
"total_ios" : 1812908,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 12924,
"mean" : 173.489903,
"stddev" : 32.365104,
"N" : 1812908
},
"clat_ns" : {
"min" : 5941,
"max" : 118202,
"mean" : 7276.558786,
"stddev" : 1678.838912,
"N" : 1812909,
"percentile" : {
"1.000000" : 6560,
"5.000000" : 6688,
"10.000000" : 6688,
"20.000000" : 6752,
"30.000000" : 6752,
"40.000000" : 6816,
"50.000000" : 6880,
"60.000000" : 6880,
"70.000000" : 6944,
"80.000000" : 7200,
"90.000000" : 8256,
"95.000000" : 8384,
"99.000000" : 14656,
"99.500000" : 18560,
"99.900000" : 20352,
"99.950000" : 21120,
"99.990000" : 62720
}
},
"lat_ns" : {
"min" : 6111,
"max" : 118372,
"mean" : 7450.048687,
"stddev" : 1675.358061,
"N" : 1812909
},
"bw_min" : 128981,
"bw_max" : 130259,
"bw_agg" : 100.000000,
"bw_mean" : 129499.928571,
"bw_dev" : 474.433989,
"bw_samples" : 14,
"iops_min" : 257963,
"iops_max" : 260518,
"iops_mean" : 258999.928571,
"iops_stddev" : 948.783875,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 32.957143,
"sys_cpu" : 67.014286,
"ctx" : 11,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 8.368213,
"2" : 91.631787,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 97.055173,
"20" : 2.794626,
"50" : 0.126316,
"100" : 0.023608,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=7252: Wed Oct 26 15:09:59 2022
read: IOPS=259k, BW=126MiB/s (133MB/s)(885MiB/7001msec)
slat (nsec): min=110, max=12924, avg=173.49, stdev=32.37
clat (usec): min=5, max=118, avg= 7.28, stdev= 1.68
lat (usec): min=6, max=118, avg= 7.45, stdev= 1.68
clat percentiles (nsec):
| 1.00th=[ 6560], 5.00th=[ 6688], 10.00th=[ 6688], 20.00th=[ 6752],
| 30.00th=[ 6752], 40.00th=[ 6816], 50.00th=[ 6880], 60.00th=[ 6880],
| 70.00th=[ 6944], 80.00th=[ 7200], 90.00th=[ 8256], 95.00th=[ 8384],
| 99.00th=[14656], 99.50th=[18560], 99.90th=[20352], 99.95th=[21120],
| 99.99th=[62720]
bw ( KiB/s): min=128981, max=130259, per=100.00%, avg=129499.93, stdev=474.43, samples=14
iops : min=257963, max=260518, avg=258999.93, stdev=948.78, samples=14
lat (usec) : 10=97.06%, 20=2.79%, 50=0.13%, 100=0.02%, 250=0.01%
cpu : usr=32.96%, sys=67.01%, ctx=11, majf=0, minf=0
IO depths : 1=8.4%, 2=91.6%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1812908,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=126MiB/s (133MB/s), 126MiB/s-126MiB/s (133MB/s-133MB/s), io=885MiB (928MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789814,
"timestamp_ms" : 1666789814565,
"time" : "Wed Oct 26 15:10:14 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 919181824,
"io_kbytes" : 897638,
"bw_bytes" : 131292933,
"bw" : 128215,
"iops" : 256431.366948,
"runtime" : 7001,
"total_ios" : 1795276,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 11972,
"mean" : 171.000074,
"stddev" : 44.899254,
"N" : 1795276
},
"clat_ns" : {
"min" : 5971,
"max" : 119064,
"mean" : 7345.391732,
"stddev" : 1661.364011,
"N" : 1795277,
"percentile" : {
"1.000000" : 6624,
"5.000000" : 6688,
"10.000000" : 6752,
"20.000000" : 6816,
"30.000000" : 6816,
"40.000000" : 6880,
"50.000000" : 6880,
"60.000000" : 6944,
"70.000000" : 7008,
"80.000000" : 7840,
"90.000000" : 8256,
"95.000000" : 8512,
"99.000000" : 14784,
"99.500000" : 18560,
"99.900000" : 20096,
"99.950000" : 20864,
"99.990000" : 62720
}
},
"lat_ns" : {
"min" : 6151,
"max" : 119244,
"mean" : 7516.391811,
"stddev" : 1658.455431,
"N" : 1795277
},
"bw_min" : 127602,
"bw_max" : 128753,
"bw_agg" : 100.000000,
"bw_mean" : 128239.071429,
"bw_dev" : 367.154225,
"bw_samples" : 14,
"iops_min" : 255204,
"iops_max" : 257506,
"iops_mean" : 256478.214286,
"iops_stddev" : 734.226771,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 33.300000,
"sys_cpu" : 66.671429,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 9.490184,
"2" : 90.509816,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 97.017896,
"20" : 2.866412,
"50" : 0.094359,
"100" : 0.021055,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=7309: Wed Oct 26 15:10:14 2022
read: IOPS=256k, BW=125MiB/s (131MB/s)(877MiB/7001msec)
slat (nsec): min=110, max=11972, avg=171.00, stdev=44.90
clat (usec): min=5, max=119, avg= 7.35, stdev= 1.66
lat (usec): min=6, max=119, avg= 7.52, stdev= 1.66
clat percentiles (nsec):
| 1.00th=[ 6624], 5.00th=[ 6688], 10.00th=[ 6752], 20.00th=[ 6816],
| 30.00th=[ 6816], 40.00th=[ 6880], 50.00th=[ 6880], 60.00th=[ 6944],
| 70.00th=[ 7008], 80.00th=[ 7840], 90.00th=[ 8256], 95.00th=[ 8512],
| 99.00th=[14784], 99.50th=[18560], 99.90th=[20096], 99.95th=[20864],
| 99.99th=[62720]
bw ( KiB/s): min=127602, max=128753, per=100.00%, avg=128239.07, stdev=367.15, samples=14
iops : min=255204, max=257506, avg=256478.21, stdev=734.23, samples=14
lat (usec) : 10=97.02%, 20=2.87%, 50=0.09%, 100=0.02%, 250=0.01%
cpu : usr=33.30%, sys=66.67%, ctx=10, majf=0, minf=0
IO depths : 1=9.5%, 2=90.5%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1795276,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=877MiB (919MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789829,
"timestamp_ms" : 1666789829766,
"time" : "Wed Oct 26 15:10:29 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 924888576,
"io_kbytes" : 903211,
"bw_bytes" : 132108066,
"bw" : 129011,
"iops" : 258023.425225,
"runtime" : 7001,
"total_ios" : 1806422,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 11772,
"mean" : 169.994528,
"stddev" : 41.933383,
"N" : 1806422
},
"clat_ns" : {
"min" : 5952,
"max" : 102713,
"mean" : 7298.990791,
"stddev" : 1647.972135,
"N" : 1806422,
"percentile" : {
"1.000000" : 6624,
"5.000000" : 6688,
"10.000000" : 6752,
"20.000000" : 6752,
"30.000000" : 6816,
"40.000000" : 6816,
"50.000000" : 6880,
"60.000000" : 6880,
"70.000000" : 6944,
"80.000000" : 7264,
"90.000000" : 8256,
"95.000000" : 8384,
"99.000000" : 14784,
"99.500000" : 18560,
"99.900000" : 20352,
"99.950000" : 20864,
"99.990000" : 62208
}
},
"lat_ns" : {
"min" : 6122,
"max" : 102883,
"mean" : 7468.985319,
"stddev" : 1644.499905,
"N" : 1806422
},
"bw_min" : 125789,
"bw_max" : 129837,
"bw_agg" : 100.000000,
"bw_mean" : 129033.714286,
"bw_dev" : 1030.428994,
"bw_samples" : 14,
"iops_min" : 251579,
"iops_max" : 259674,
"iops_mean" : 258067.500000,
"iops_stddev" : 2060.615769,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 36.842857,
"sys_cpu" : 63.128571,
"ctx" : 16,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 8.782610,
"2" : 91.217390,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 97.043991,
"20" : 2.821212,
"50" : 0.114259,
"100" : 0.020427,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=7367: Wed Oct 26 15:10:29 2022
read: IOPS=258k, BW=126MiB/s (132MB/s)(882MiB/7001msec)
slat (nsec): min=110, max=11772, avg=169.99, stdev=41.93
clat (usec): min=5, max=102, avg= 7.30, stdev= 1.65
lat (usec): min=6, max=102, avg= 7.47, stdev= 1.64
clat percentiles (nsec):
| 1.00th=[ 6624], 5.00th=[ 6688], 10.00th=[ 6752], 20.00th=[ 6752],
| 30.00th=[ 6816], 40.00th=[ 6816], 50.00th=[ 6880], 60.00th=[ 6880],
| 70.00th=[ 6944], 80.00th=[ 7264], 90.00th=[ 8256], 95.00th=[ 8384],
| 99.00th=[14784], 99.50th=[18560], 99.90th=[20352], 99.95th=[20864],
| 99.99th=[62208]
bw ( KiB/s): min=125789, max=129837, per=100.00%, avg=129033.71, stdev=1030.43, samples=14
iops : min=251579, max=259674, avg=258067.50, stdev=2060.62, samples=14
lat (usec) : 10=97.04%, 20=2.82%, 50=0.11%, 100=0.02%, 250=0.01%
cpu : usr=36.84%, sys=63.13%, ctx=16, majf=0, minf=0
IO depths : 1=8.8%, 2=91.2%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1806422,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=882MiB (925MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789845,
"timestamp_ms" : 1666789845262,
"time" : "Wed Oct 26 15:10:45 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1071219200,
"io_kbytes" : 1046112,
"bw_bytes" : 153009455,
"bw" : 149423,
"iops" : 298846.307670,
"runtime" : 7001,
"total_ios" : 2092223,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1683,
"max" : 35506,
"mean" : 1886.948066,
"stddev" : 122.911174,
"N" : 2092224
},
"clat_ns" : {
"min" : 260,
"max" : 119775,
"mean" : 4572.451873,
"stddev" : 1788.671868,
"N" : 2092224,
"percentile" : {
"1.000000" : 3888,
"5.000000" : 4128,
"10.000000" : 4192,
"20.000000" : 4256,
"30.000000" : 4256,
"40.000000" : 4320,
"50.000000" : 4320,
"60.000000" : 4320,
"70.000000" : 4384,
"80.000000" : 4448,
"90.000000" : 4448,
"95.000000" : 4640,
"99.000000" : 12608,
"99.500000" : 17536,
"99.900000" : 21888,
"99.950000" : 22144,
"99.990000" : 63232
}
},
"lat_ns" : {
"min" : 5139,
"max" : 121608,
"mean" : 6459.399939,
"stddev" : 1792.133017,
"N" : 2092224
},
"bw_min" : 148978,
"bw_max" : 149936,
"bw_agg" : 100.000000,
"bw_mean" : 149450.785714,
"bw_dev" : 305.810695,
"bw_samples" : 14,
"iops_min" : 297956,
"iops_max" : 299873,
"iops_mean" : 298901.642857,
"iops_stddev" : 612.123116,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 44.157143,
"sys_cpu" : 55.828571,
"ctx" : 13,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 3.109707,
"10" : 94.906518,
"20" : 1.720562,
"50" : 0.241657,
"100" : 0.020982,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=7424: Wed Oct 26 15:10:45 2022
read: IOPS=299k, BW=146MiB/s (153MB/s)(1022MiB/7001msec)
slat (nsec): min=1683, max=35506, avg=1886.95, stdev=122.91
clat (nsec): min=260, max=119775, avg=4572.45, stdev=1788.67
lat (usec): min=5, max=121, avg= 6.46, stdev= 1.79
clat percentiles (nsec):
| 1.00th=[ 3888], 5.00th=[ 4128], 10.00th=[ 4192], 20.00th=[ 4256],
| 30.00th=[ 4256], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4320],
| 70.00th=[ 4384], 80.00th=[ 4448], 90.00th=[ 4448], 95.00th=[ 4640],
| 99.00th=[12608], 99.50th=[17536], 99.90th=[21888], 99.95th=[22144],
| 99.99th=[63232]
bw ( KiB/s): min=148978, max=149936, per=100.00%, avg=149450.79, stdev=305.81, samples=14
iops : min=297956, max=299873, avg=298901.64, stdev=612.12, samples=14
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 4=3.11%, 10=94.91%, 20=1.72%, 50=0.24%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=44.16%, sys=55.83%, ctx=13, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2092223,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=1022MiB (1071MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789203,
"timestamp_ms" : 1666789203700,
"time" : "Wed Oct 26 15:00:03 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 394408960,
"io_kbytes" : 385165,
"bw_bytes" : 56336089,
"bw" : 55015,
"iops" : 110031.424082,
"runtime" : 7001,
"total_ios" : 770330,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1402,
"max" : 14557,
"mean" : 1512.964761,
"stddev" : 62.706972,
"N" : 770330
},
"clat_ns" : {
"min" : 260,
"max" : 97473,
"mean" : 7280.894260,
"stddev" : 982.896349,
"N" : 770330,
"percentile" : {
"1.000000" : 6752,
"5.000000" : 7008,
"10.000000" : 7072,
"20.000000" : 7072,
"30.000000" : 7072,
"40.000000" : 7136,
"50.000000" : 7200,
"60.000000" : 7200,
"70.000000" : 7264,
"80.000000" : 7264,
"90.000000" : 7392,
"95.000000" : 7456,
"99.000000" : 10432,
"99.500000" : 13632,
"99.900000" : 15680,
"99.950000" : 15808,
"99.990000" : 58112
}
},
"lat_ns" : {
"min" : 6993,
"max" : 98975,
"mean" : 8793.859020,
"stddev" : 985.772717,
"N" : 770330
},
"bw_min" : 54788,
"bw_max" : 55269,
"bw_agg" : 100.000000,
"bw_mean" : 55025.500000,
"bw_dev" : 130.894408,
"bw_samples" : 14,
"iops_min" : 109576,
"iops_max" : 110538,
"iops_mean" : 110051.000000,
"iops_stddev" : 261.788817,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 8.414286,
"sys_cpu" : 35.785714,
"ctx" : 770332,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 98.690691,
"20" : 1.290486,
"50" : 0.010000,
"100" : 0.014799,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=5122: Wed Oct 26 15:00:03 2022
read: IOPS=110k, BW=53.7MiB/s (56.3MB/s)(376MiB/7001msec)
slat (nsec): min=1402, max=14557, avg=1512.96, stdev=62.71
clat (nsec): min=260, max=97473, avg=7280.89, stdev=982.90
lat (nsec): min=6993, max=98975, avg=8793.86, stdev=985.77
clat percentiles (nsec):
| 1.00th=[ 6752], 5.00th=[ 7008], 10.00th=[ 7072], 20.00th=[ 7072],
| 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7200], 60.00th=[ 7200],
| 70.00th=[ 7264], 80.00th=[ 7264], 90.00th=[ 7392], 95.00th=[ 7456],
| 99.00th=[10432], 99.50th=[13632], 99.90th=[15680], 99.95th=[15808],
| 99.99th=[58112]
bw ( KiB/s): min=54788, max=55269, per=100.00%, avg=55025.50, stdev=130.89, samples=14
iops : min=109576, max=110538, avg=110051.00, stdev=261.79, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=98.69%, 20=1.29%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=8.41%, sys=35.79%, ctx=770332, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=770330,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=53.7MiB/s (56.3MB/s), 53.7MiB/s-53.7MiB/s (56.3MB/s-56.3MB/s), io=376MiB (394MB), run=7001-7001msec
$ cat /tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789860,
"timestamp_ms" : 1666789860552,
"time" : "Wed Oct 26 15:11:00 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1074551808,
"io_kbytes" : 1049367,
"bw_bytes" : 153485474,
"bw" : 149888,
"iops" : 299776.174832,
"runtime" : 7001,
"total_ios" : 2098733,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1673,
"max" : 23223,
"mean" : 1867.469730,
"stddev" : 122.869433,
"N" : 2098733
},
"clat_ns" : {
"min" : 230,
"max" : 115426,
"mean" : 4579.960121,
"stddev" : 1762.455468,
"N" : 2098734,
"percentile" : {
"1.000000" : 3888,
"5.000000" : 4128,
"10.000000" : 4192,
"20.000000" : 4256,
"30.000000" : 4256,
"40.000000" : 4320,
"50.000000" : 4320,
"60.000000" : 4384,
"70.000000" : 4384,
"80.000000" : 4448,
"90.000000" : 4512,
"95.000000" : 4640,
"99.000000" : 12608,
"99.500000" : 17536,
"99.900000" : 21888,
"99.950000" : 22400,
"99.990000" : 61184
}
},
"lat_ns" : {
"min" : 5070,
"max" : 117269,
"mean" : 6447.429815,
"stddev" : 1765.832177,
"N" : 2098734
},
"bw_min" : 149368,
"bw_max" : 150878,
"bw_agg" : 100.000000,
"bw_mean" : 149916.857143,
"bw_dev" : 485.029080,
"bw_samples" : 14,
"iops_min" : 298736,
"iops_max" : 301757,
"iops_mean" : 299833.785714,
"iops_stddev" : 970.210618,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 28.900000,
"sys_cpu" : 71.085714,
"ctx" : 9,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 3.019584,
"10" : 94.994837,
"20" : 1.723421,
"50" : 0.241717,
"100" : 0.019631,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=7482: Wed Oct 26 15:11:00 2022
read: IOPS=300k, BW=146MiB/s (153MB/s)(1025MiB/7001msec)
slat (nsec): min=1673, max=23223, avg=1867.47, stdev=122.87
clat (nsec): min=230, max=115426, avg=4579.96, stdev=1762.46
lat (usec): min=5, max=117, avg= 6.45, stdev= 1.77
clat percentiles (nsec):
| 1.00th=[ 3888], 5.00th=[ 4128], 10.00th=[ 4192], 20.00th=[ 4256],
| 30.00th=[ 4256], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4384],
| 70.00th=[ 4384], 80.00th=[ 4448], 90.00th=[ 4512], 95.00th=[ 4640],
| 99.00th=[12608], 99.50th=[17536], 99.90th=[21888], 99.95th=[22400],
| 99.99th=[61184]
bw ( KiB/s): min=149368, max=150878, per=100.00%, avg=149916.86, stdev=485.03, samples=14
iops : min=298736, max=301757, avg=299833.79, stdev=970.21, samples=14
lat (nsec) : 250=0.01%, 500=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=3.02%, 10=94.99%, 20=1.72%, 50=0.24%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=28.90%, sys=71.09%, ctx=9, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2098733,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=1025MiB (1075MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789875,
"timestamp_ms" : 1666789875781,
"time" : "Wed Oct 26 15:11:15 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1068184064,
"io_kbytes" : 1043148,
"bw_bytes" : 152575926,
"bw" : 148999,
"iops" : 297999.714327,
"runtime" : 7001,
"total_ios" : 2086296,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1723,
"max" : 23454,
"mean" : 1919.903935,
"stddev" : 128.714954,
"N" : 2086296
},
"clat_ns" : {
"min" : 250,
"max" : 98895,
"mean" : 4553.953938,
"stddev" : 1784.346449,
"N" : 2086297,
"percentile" : {
"1.000000" : 3856,
"5.000000" : 4128,
"10.000000" : 4192,
"20.000000" : 4192,
"30.000000" : 4256,
"40.000000" : 4256,
"50.000000" : 4320,
"60.000000" : 4320,
"70.000000" : 4384,
"80.000000" : 4384,
"90.000000" : 4448,
"95.000000" : 4640,
"99.000000" : 12608,
"99.500000" : 17536,
"99.900000" : 21888,
"99.950000" : 22144,
"99.990000" : 62720
}
},
"lat_ns" : {
"min" : 5169,
"max" : 100759,
"mean" : 6473.857861,
"stddev" : 1787.577180,
"N" : 2086297
},
"bw_min" : 148578,
"bw_max" : 149622,
"bw_agg" : 100.000000,
"bw_mean" : 149027.000000,
"bw_dev" : 264.555069,
"bw_samples" : 14,
"iops_min" : 297156,
"iops_max" : 299244,
"iops_mean" : 298054.000000,
"iops_stddev" : 529.110138,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 48.900000,
"sys_cpu" : 51.085714,
"ctx" : 15,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 3.277243,
"10" : 94.723520,
"20" : 1.731921,
"50" : 0.245459,
"100" : 0.021665,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=7539: Wed Oct 26 15:11:15 2022
read: IOPS=298k, BW=146MiB/s (153MB/s)(1019MiB/7001msec)
slat (nsec): min=1723, max=23454, avg=1919.90, stdev=128.71
clat (nsec): min=250, max=98895, avg=4553.95, stdev=1784.35
lat (usec): min=5, max=100, avg= 6.47, stdev= 1.79
clat percentiles (nsec):
| 1.00th=[ 3856], 5.00th=[ 4128], 10.00th=[ 4192], 20.00th=[ 4192],
| 30.00th=[ 4256], 40.00th=[ 4256], 50.00th=[ 4320], 60.00th=[ 4320],
| 70.00th=[ 4384], 80.00th=[ 4384], 90.00th=[ 4448], 95.00th=[ 4640],
| 99.00th=[12608], 99.50th=[17536], 99.90th=[21888], 99.95th=[22144],
| 99.99th=[62720]
bw ( KiB/s): min=148578, max=149622, per=100.00%, avg=149027.00, stdev=264.56, samples=14
iops : min=297156, max=299244, avg=298054.00, stdev=529.11, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=3.28%, 10=94.72%, 20=1.73%, 50=0.25%, 100=0.02%
cpu : usr=48.90%, sys=51.09%, ctx=15, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2086296,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=1019MiB (1068MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789889,
"timestamp_ms" : 1666789889671,
"time" : "Wed Oct 26 15:11:29 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 755842560,
"io_kbytes" : 738127,
"bw_bytes" : 107962085,
"bw" : 105431,
"iops" : 210863.305242,
"runtime" : 7001,
"total_ios" : 1476254,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1373,
"max" : 14708,
"mean" : 1626.049088,
"stddev" : 183.510630,
"N" : 1476254
},
"clat_ns" : {
"min" : 180,
"max" : 118923,
"mean" : 7596.831394,
"stddev" : 2223.151074,
"N" : 1476255,
"percentile" : {
"1.000000" : 5152,
"5.000000" : 5280,
"10.000000" : 5344,
"20.000000" : 5472,
"30.000000" : 6752,
"40.000000" : 7264,
"50.000000" : 7456,
"60.000000" : 7712,
"70.000000" : 8032,
"80.000000" : 9280,
"90.000000" : 9408,
"95.000000" : 9536,
"99.000000" : 16768,
"99.500000" : 18560,
"99.900000" : 23936,
"99.950000" : 24448,
"99.990000" : 61184
}
},
"lat_ns" : {
"min" : 5811,
"max" : 120425,
"mean" : 9222.880527,
"stddev" : 2251.499860,
"N" : 1476255
},
"bw_min" : 105209,
"bw_max" : 105680,
"bw_agg" : 100.000000,
"bw_mean" : 105450.357143,
"bw_dev" : 127.924020,
"bw_samples" : 14,
"iops_min" : 210418,
"iops_max" : 211362,
"iops_mean" : 210900.785714,
"iops_stddev" : 255.828722,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.514286,
"sys_cpu" : 51.542857,
"ctx" : 741997,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.311100,
"20" : 2.439621,
"50" : 0.215004,
"100" : 0.024522,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=7587: Wed Oct 26 15:11:29 2022
read: IOPS=211k, BW=103MiB/s (108MB/s)(721MiB/7001msec)
slat (nsec): min=1373, max=14708, avg=1626.05, stdev=183.51
clat (nsec): min=180, max=118923, avg=7596.83, stdev=2223.15
lat (usec): min=5, max=120, avg= 9.22, stdev= 2.25
clat percentiles (nsec):
| 1.00th=[ 5152], 5.00th=[ 5280], 10.00th=[ 5344], 20.00th=[ 5472],
| 30.00th=[ 6752], 40.00th=[ 7264], 50.00th=[ 7456], 60.00th=[ 7712],
| 70.00th=[ 8032], 80.00th=[ 9280], 90.00th=[ 9408], 95.00th=[ 9536],
| 99.00th=[16768], 99.50th=[18560], 99.90th=[23936], 99.95th=[24448],
| 99.99th=[61184]
bw ( KiB/s): min=105209, max=105680, per=100.00%, avg=105450.36, stdev=127.92, samples=14
iops : min=210418, max=211362, avg=210900.79, stdev=255.83, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=97.31%, 20=2.44%, 50=0.22%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=14.51%, sys=51.54%, ctx=741997, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1476254,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=721MiB (756MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789903,
"timestamp_ms" : 1666789903447,
"time" : "Wed Oct 26 15:11:43 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 751161856,
"io_kbytes" : 733556,
"bw_bytes" : 107293508,
"bw" : 104778,
"iops" : 209557.491787,
"runtime" : 7001,
"total_ios" : 1467112,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1402,
"max" : 30337,
"mean" : 1675.573352,
"stddev" : 230.937066,
"N" : 1467112
},
"clat_ns" : {
"min" : 2745,
"max" : 119153,
"mean" : 7591.607664,
"stddev" : 2285.968158,
"N" : 1467113,
"percentile" : {
"1.000000" : 5216,
"5.000000" : 5280,
"10.000000" : 5344,
"20.000000" : 5472,
"30.000000" : 5984,
"40.000000" : 7136,
"50.000000" : 7456,
"60.000000" : 7712,
"70.000000" : 8160,
"80.000000" : 9408,
"90.000000" : 9664,
"95.000000" : 9792,
"99.000000" : 16768,
"99.500000" : 18560,
"99.900000" : 24192,
"99.950000" : 24448,
"99.990000" : 64256
}
},
"lat_ns" : {
"min" : 5801,
"max" : 120656,
"mean" : 9267.180865,
"stddev" : 2311.110610,
"N" : 1467113
},
"bw_min" : 104352,
"bw_max" : 105176,
"bw_agg" : 100.000000,
"bw_mean" : 104797.642857,
"bw_dev" : 211.042689,
"bw_samples" : 14,
"iops_min" : 208705,
"iops_max" : 210352,
"iops_mean" : 209595.357143,
"iops_stddev" : 421.922998,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.942857,
"sys_cpu" : 51.414286,
"ctx" : 737441,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010156,
"10" : 97.018701,
"20" : 2.717039,
"50" : 0.231816,
"100" : 0.022016,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=7635: Wed Oct 26 15:11:43 2022
read: IOPS=210k, BW=102MiB/s (107MB/s)(716MiB/7001msec)
slat (nsec): min=1402, max=30337, avg=1675.57, stdev=230.94
clat (usec): min=2, max=119, avg= 7.59, stdev= 2.29
lat (usec): min=5, max=120, avg= 9.27, stdev= 2.31
clat percentiles (nsec):
| 1.00th=[ 5216], 5.00th=[ 5280], 10.00th=[ 5344], 20.00th=[ 5472],
| 30.00th=[ 5984], 40.00th=[ 7136], 50.00th=[ 7456], 60.00th=[ 7712],
| 70.00th=[ 8160], 80.00th=[ 9408], 90.00th=[ 9664], 95.00th=[ 9792],
| 99.00th=[16768], 99.50th=[18560], 99.90th=[24192], 99.95th=[24448],
| 99.99th=[64256]
bw ( KiB/s): min=104352, max=105176, per=100.00%, avg=104797.64, stdev=211.04, samples=14
iops : min=208705, max=210352, avg=209595.36, stdev=421.92, samples=14
lat (usec) : 4=0.01%, 10=97.02%, 20=2.72%, 50=0.23%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=15.94%, sys=51.41%, ctx=737441, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1467112,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=102MiB/s (107MB/s), 102MiB/s-102MiB/s (107MB/s-107MB/s), io=716MiB (751MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt': No such file or directory
$ rm /tmp/fio-output_512_2_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789917,
"timestamp_ms" : 1666789917192,
"time" : "Wed Oct 26 15:11:57 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 754146304,
"io_kbytes" : 736471,
"bw_bytes" : 107735186,
"bw" : 105210,
"iops" : 210420.142857,
"runtime" : 7000,
"total_ios" : 1472941,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1373,
"max" : 39404,
"mean" : 1621.156412,
"stddev" : 192.150453,
"N" : 1472941
},
"clat_ns" : {
"min" : 180,
"max" : 110587,
"mean" : 7626.157111,
"stddev" : 2228.400265,
"N" : 1472942,
"percentile" : {
"1.000000" : 5216,
"5.000000" : 5280,
"10.000000" : 5344,
"20.000000" : 5472,
"30.000000" : 6752,
"40.000000" : 7264,
"50.000000" : 7456,
"60.000000" : 7712,
"70.000000" : 8096,
"80.000000" : 9280,
"90.000000" : 9408,
"95.000000" : 9664,
"99.000000" : 16768,
"99.500000" : 18560,
"99.900000" : 23936,
"99.950000" : 24448,
"99.990000" : 61696
}
},
"lat_ns" : {
"min" : 5741,
"max" : 112090,
"mean" : 9247.313388,
"stddev" : 2260.379617,
"N" : 1472942
},
"bw_min" : 104853,
"bw_max" : 105510,
"bw_agg" : 100.000000,
"bw_mean" : 105237.769231,
"bw_dev" : 207.918716,
"bw_samples" : 13,
"iops_min" : 209706,
"iops_max" : 211020,
"iops_mean" : 210475.769231,
"iops_stddev" : 415.849162,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.157143,
"sys_cpu" : 51.885714,
"ctx" : 740379,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.277556,
"20" : 2.467716,
"50" : 0.220783,
"100" : 0.024101,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=7683: Wed Oct 26 15:11:57 2022
read: IOPS=210k, BW=103MiB/s (108MB/s)(719MiB/7000msec)
slat (nsec): min=1373, max=39404, avg=1621.16, stdev=192.15
clat (nsec): min=180, max=110587, avg=7626.16, stdev=2228.40
lat (usec): min=5, max=112, avg= 9.25, stdev= 2.26
clat percentiles (nsec):
| 1.00th=[ 5216], 5.00th=[ 5280], 10.00th=[ 5344], 20.00th=[ 5472],
| 30.00th=[ 6752], 40.00th=[ 7264], 50.00th=[ 7456], 60.00th=[ 7712],
| 70.00th=[ 8096], 80.00th=[ 9280], 90.00th=[ 9408], 95.00th=[ 9664],
| 99.00th=[16768], 99.50th=[18560], 99.90th=[23936], 99.95th=[24448],
| 99.99th=[61696]
bw ( KiB/s): min=104853, max=105510, per=100.00%, avg=105237.77, stdev=207.92, samples=13
iops : min=209706, max=211020, avg=210475.77, stdev=415.85, samples=13
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=97.28%, 20=2.47%, 50=0.22%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=14.16%, sys=51.89%, ctx=740379, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1472941,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=719MiB (754MB), run=7000-7000msec
$ rm /tmp/fio-output_512_2_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789931,
"timestamp_ms" : 1666789931060,
"time" : "Wed Oct 26 15:12:11 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 799335936,
"io_kbytes" : 780601,
"bw_bytes" : 114174537,
"bw" : 111498,
"iops" : 222997.000429,
"runtime" : 7001,
"total_ios" : 1561202,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1442,
"max" : 21661,
"mean" : 1649.153308,
"stddev" : 129.768110,
"N" : 1561202
},
"clat_ns" : {
"min" : 2645,
"max" : 100528,
"mean" : 7067.635010,
"stddev" : 1738.217242,
"N" : 1561203,
"percentile" : {
"1.000000" : 5856,
"5.000000" : 5920,
"10.000000" : 5984,
"20.000000" : 6112,
"30.000000" : 6880,
"40.000000" : 6944,
"50.000000" : 7008,
"60.000000" : 7136,
"70.000000" : 7200,
"80.000000" : 7264,
"90.000000" : 7392,
"95.000000" : 7520,
"99.000000" : 15936,
"99.500000" : 18560,
"99.900000" : 25216,
"99.950000" : 25728,
"99.990000" : 62720
}
},
"lat_ns" : {
"min" : 6422,
"max" : 102221,
"mean" : 8716.788250,
"stddev" : 1739.886131,
"N" : 1561203
},
"bw_min" : 110982,
"bw_max" : 112688,
"bw_agg" : 100.000000,
"bw_mean" : 111528.500000,
"bw_dev" : 424.069797,
"bw_samples" : 14,
"iops_min" : 221965,
"iops_max" : 225376,
"iops_mean" : 223057.071429,
"iops_stddev" : 848.040498,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 20.328571,
"sys_cpu" : 47.114286,
"ctx" : 783978,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.719770,
"20" : 1.871763,
"50" : 0.379771,
"100" : 0.021009,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2224647,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 10878,
"write_ticks" : 0,
"in_queue" : 10878,
"util" : 99.069123
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=7730: Wed Oct 26 15:12:11 2022
read: IOPS=223k, BW=109MiB/s (114MB/s)(762MiB/7001msec)
slat (nsec): min=1442, max=21661, avg=1649.15, stdev=129.77
clat (usec): min=2, max=100, avg= 7.07, stdev= 1.74
lat (usec): min=6, max=102, avg= 8.72, stdev= 1.74
clat percentiles (nsec):
| 1.00th=[ 5856], 5.00th=[ 5920], 10.00th=[ 5984], 20.00th=[ 6112],
| 30.00th=[ 6880], 40.00th=[ 6944], 50.00th=[ 7008], 60.00th=[ 7136],
| 70.00th=[ 7200], 80.00th=[ 7264], 90.00th=[ 7392], 95.00th=[ 7520],
| 99.00th=[15936], 99.50th=[18560], 99.90th=[25216], 99.95th=[25728],
| 99.99th=[62720]
bw ( KiB/s): min=110982, max=112688, per=100.00%, avg=111528.50, stdev=424.07, samples=14
iops : min=221965, max=225376, avg=223057.07, stdev=848.04, samples=14
lat (usec) : 4=0.01%, 10=97.72%, 20=1.87%, 50=0.38%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=20.33%, sys=47.11%, ctx=783978, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1561202,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=109MiB/s (114MB/s), 109MiB/s-109MiB/s (114MB/s-114MB/s), io=762MiB (799MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2224647/0, merge=0/0, ticks=10878/0, in_queue=10878, util=99.07%
$ rm /tmp/fio-output_512_2_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789944,
"timestamp_ms" : 1666789944855,
"time" : "Wed Oct 26 15:12:24 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 815441920,
"io_kbytes" : 796330,
"bw_bytes" : 116475063,
"bw" : 113745,
"iops" : 227490.072847,
"runtime" : 7001,
"total_ios" : 1592658,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1472,
"max" : 20098,
"mean" : 1679.534212,
"stddev" : 135.170532,
"N" : 1592659
},
"clat_ns" : {
"min" : 431,
"max" : 91642,
"mean" : 6852.600837,
"stddev" : 1799.750306,
"N" : 1592659,
"percentile" : {
"1.000000" : 5856,
"5.000000" : 5920,
"10.000000" : 5920,
"20.000000" : 6048,
"30.000000" : 6112,
"40.000000" : 6240,
"50.000000" : 6368,
"60.000000" : 7008,
"70.000000" : 7136,
"80.000000" : 7328,
"90.000000" : 7456,
"95.000000" : 7584,
"99.000000" : 15680,
"99.500000" : 18560,
"99.900000" : 25472,
"99.950000" : 25984,
"99.990000" : 62720
}
},
"lat_ns" : {
"min" : 5681,
"max" : 93386,
"mean" : 8532.135048,
"stddev" : 1803.334792,
"N" : 1592659
},
"bw_min" : 113374,
"bw_max" : 114111,
"bw_agg" : 100.000000,
"bw_mean" : 113776.000000,
"bw_dev" : 230.273082,
"bw_samples" : 14,
"iops_min" : 226748,
"iops_max" : 228222,
"iops_mean" : 227551.928571,
"iops_stddev" : 460.425633,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 23.000000,
"sys_cpu" : 47.142857,
"ctx" : 800025,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.734856,
"20" : 1.887411,
"50" : 0.346276,
"100" : 0.023420,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2265813,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 11215,
"write_ticks" : 0,
"in_queue" : 11215,
"util" : 99.069123
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=7778: Wed Oct 26 15:12:24 2022
read: IOPS=227k, BW=111MiB/s (116MB/s)(778MiB/7001msec)
slat (nsec): min=1472, max=20098, avg=1679.53, stdev=135.17
clat (nsec): min=431, max=91642, avg=6852.60, stdev=1799.75
lat (nsec): min=5681, max=93386, avg=8532.14, stdev=1803.33
clat percentiles (nsec):
| 1.00th=[ 5856], 5.00th=[ 5920], 10.00th=[ 5920], 20.00th=[ 6048],
| 30.00th=[ 6112], 40.00th=[ 6240], 50.00th=[ 6368], 60.00th=[ 7008],
| 70.00th=[ 7136], 80.00th=[ 7328], 90.00th=[ 7456], 95.00th=[ 7584],
| 99.00th=[15680], 99.50th=[18560], 99.90th=[25472], 99.95th=[25984],
| 99.99th=[62720]
bw ( KiB/s): min=113374, max=114111, per=100.00%, avg=113776.00, stdev=230.27, samples=14
iops : min=226748, max=228222, avg=227551.93, stdev=460.43, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=97.73%, 20=1.89%, 50=0.35%, 100=0.02%
cpu : usr=23.00%, sys=47.14%, ctx=800025, majf=0, minf=1
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1592658,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=111MiB/s (116MB/s), 111MiB/s-111MiB/s (116MB/s-116MB/s), io=778MiB (815MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2265813/0, merge=0/0, ticks=11215/0, in_queue=11215, util=99.07%
$ rm /tmp/fio-output_512_2_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789958,
"timestamp_ms" : 1666789958677,
"time" : "Wed Oct 26 15:12:38 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 812047872,
"io_kbytes" : 793015,
"bw_bytes" : 115990268,
"bw" : 113271,
"iops" : 226543.208113,
"runtime" : 7001,
"total_ios" : 1586029,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1452,
"max" : 14798,
"mean" : 1655.304542,
"stddev" : 123.996712,
"N" : 1586030
},
"clat_ns" : {
"min" : 421,
"max" : 120085,
"mean" : 6916.411781,
"stddev" : 1733.488478,
"N" : 1586030,
"percentile" : {
"1.000000" : 5792,
"5.000000" : 5856,
"10.000000" : 5920,
"20.000000" : 6048,
"30.000000" : 6112,
"40.000000" : 6816,
"50.000000" : 6944,
"60.000000" : 7008,
"70.000000" : 7136,
"80.000000" : 7264,
"90.000000" : 7328,
"95.000000" : 7520,
"99.000000" : 15680,
"99.500000" : 18560,
"99.900000" : 24192,
"99.950000" : 25728,
"99.990000" : 61184
}
},
"lat_ns" : {
"min" : 5661,
"max" : 121798,
"mean" : 8571.716323,
"stddev" : 1737.367517,
"N" : 1586030
},
"bw_min" : 112211,
"bw_max" : 114174,
"bw_agg" : 100.000000,
"bw_mean" : 113343.571429,
"bw_dev" : 546.633292,
"bw_samples" : 14,
"iops_min" : 224422,
"iops_max" : 228348,
"iops_mean" : 226687.285714,
"iops_stddev" : 1093.360904,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 21.314286,
"sys_cpu" : 47.257143,
"ctx" : 796376,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.729802,
"20" : 1.891264,
"50" : 0.351885,
"100" : 0.018789,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2250040,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 11049,
"write_ticks" : 0,
"in_queue" : 11049,
"util" : 99.108381
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=7824: Wed Oct 26 15:12:38 2022
read: IOPS=227k, BW=111MiB/s (116MB/s)(774MiB/7001msec)
slat (nsec): min=1452, max=14798, avg=1655.30, stdev=124.00
clat (nsec): min=421, max=120085, avg=6916.41, stdev=1733.49
lat (usec): min=5, max=121, avg= 8.57, stdev= 1.74
clat percentiles (nsec):
| 1.00th=[ 5792], 5.00th=[ 5856], 10.00th=[ 5920], 20.00th=[ 6048],
| 30.00th=[ 6112], 40.00th=[ 6816], 50.00th=[ 6944], 60.00th=[ 7008],
| 70.00th=[ 7136], 80.00th=[ 7264], 90.00th=[ 7328], 95.00th=[ 7520],
| 99.00th=[15680], 99.50th=[18560], 99.90th=[24192], 99.95th=[25728],
| 99.99th=[61184]
bw ( KiB/s): min=112211, max=114174, per=100.00%, avg=113343.57, stdev=546.63, samples=14
iops : min=224422, max=228348, avg=226687.29, stdev=1093.36, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=97.73%, 20=1.89%, 50=0.35%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=21.31%, sys=47.26%, ctx=796376, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1586029,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=111MiB/s (116MB/s), 111MiB/s-111MiB/s (116MB/s-116MB/s), io=774MiB (812MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2250040/0, merge=0/0, ticks=11049/0, in_queue=11049, util=99.11%
$ rm /tmp/fio-output_512_2_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-bdev_aio_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789974,
"timestamp_ms" : 1666789974420,
"time" : "Wed Oct 26 15:12:54 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1065744896,
"io_kbytes" : 1040766,
"bw_bytes" : 152227524,
"bw" : 148659,
"iops" : 297319.240109,
"runtime" : 7001,
"total_ios" : 2081532,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1663,
"max" : 22833,
"mean" : 1889.934944,
"stddev" : 127.676401,
"N" : 2081532
},
"clat_ns" : {
"min" : 241,
"max" : 102011,
"mean" : 4605.353980,
"stddev" : 1741.728440,
"N" : 2081533,
"percentile" : {
"1.000000" : 3952,
"5.000000" : 4192,
"10.000000" : 4256,
"20.000000" : 4256,
"30.000000" : 4320,
"40.000000" : 4320,
"50.000000" : 4384,
"60.000000" : 4384,
"70.000000" : 4384,
"80.000000" : 4448,
"90.000000" : 4512,
"95.000000" : 4704,
"99.000000" : 12608,
"99.500000" : 17536,
"99.900000" : 21888,
"99.950000" : 22400,
"99.990000" : 59648
}
},
"lat_ns" : {
"min" : 5129,
"max" : 103885,
"mean" : 6495.288902,
"stddev" : 1745.563229,
"N" : 2081533
},
"bw_min" : 147611,
"bw_max" : 149598,
"bw_agg" : 100.000000,
"bw_mean" : 148687.142857,
"bw_dev" : 617.469882,
"bw_samples" : 14,
"iops_min" : 295222,
"iops_max" : 299196,
"iops_mean" : 297374.357143,
"iops_stddev" : 1234.727102,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 63.985714,
"sys_cpu" : 35.985714,
"ctx" : 15,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 99.989959,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 2.077076,
"10" : 95.930641,
"20" : 1.729688,
"50" : 0.242177,
"100" : 0.017968,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=7881: Wed Oct 26 15:12:54 2022
read: IOPS=297k, BW=145MiB/s (152MB/s)(1016MiB/7001msec)
slat (nsec): min=1663, max=22833, avg=1889.93, stdev=127.68
clat (nsec): min=241, max=102011, avg=4605.35, stdev=1741.73
lat (usec): min=5, max=103, avg= 6.50, stdev= 1.75
clat percentiles (nsec):
| 1.00th=[ 3952], 5.00th=[ 4192], 10.00th=[ 4256], 20.00th=[ 4256],
| 30.00th=[ 4320], 40.00th=[ 4320], 50.00th=[ 4384], 60.00th=[ 4384],
| 70.00th=[ 4384], 80.00th=[ 4448], 90.00th=[ 4512], 95.00th=[ 4704],
| 99.00th=[12608], 99.50th=[17536], 99.90th=[21888], 99.95th=[22400],
| 99.99th=[59648]
bw ( KiB/s): min=147611, max=149598, per=100.00%, avg=148687.14, stdev=617.47, samples=14
iops : min=295222, max=299196, avg=297374.36, stdev=1234.73, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=2.08%, 10=95.93%, 20=1.73%, 50=0.24%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=63.99%, sys=35.99%, ctx=15, majf=0, minf=0
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2081532,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=1016MiB (1066MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-bdev_aio_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789989,
"timestamp_ms" : 1666789989718,
"time" : "Wed Oct 26 15:13:09 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1061161472,
"io_kbytes" : 1036290,
"bw_bytes" : 151572842,
"bw" : 148020,
"iops" : 296040.422797,
"runtime" : 7001,
"total_ios" : 2072579,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1683,
"max" : 35486,
"mean" : 1927.547332,
"stddev" : 125.332926,
"N" : 2072580
},
"clat_ns" : {
"min" : 240,
"max" : 98986,
"mean" : 4604.571840,
"stddev" : 1756.957642,
"N" : 2072579,
"percentile" : {
"1.000000" : 3952,
"5.000000" : 4192,
"10.000000" : 4256,
"20.000000" : 4256,
"30.000000" : 4320,
"40.000000" : 4320,
"50.000000" : 4384,
"60.000000" : 4384,
"70.000000" : 4384,
"80.000000" : 4448,
"90.000000" : 4512,
"95.000000" : 4704,
"99.000000" : 12608,
"99.500000" : 17536,
"99.900000" : 21888,
"99.950000" : 22400,
"99.990000" : 61184
}
},
"lat_ns" : {
"min" : 5180,
"max" : 100919,
"mean" : 6532.119198,
"stddev" : 1759.977965,
"N" : 2072579
},
"bw_min" : 147349,
"bw_max" : 148689,
"bw_agg" : 100.000000,
"bw_mean" : 148046.357143,
"bw_dev" : 438.091070,
"bw_samples" : 14,
"iops_min" : 294698,
"iops_max" : 297378,
"iops_mean" : 296092.714286,
"iops_stddev" : 876.182140,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 63.528571,
"sys_cpu" : 36.442857,
"ctx" : 11,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 99.989240,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 2.120884,
"10" : 95.873450,
"20" : 1.738317,
"50" : 0.244237,
"100" : 0.020506,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=7940: Wed Oct 26 15:13:09 2022
read: IOPS=296k, BW=145MiB/s (152MB/s)(1012MiB/7001msec)
slat (nsec): min=1683, max=35486, avg=1927.55, stdev=125.33
clat (nsec): min=240, max=98986, avg=4604.57, stdev=1756.96
lat (usec): min=5, max=100, avg= 6.53, stdev= 1.76
clat percentiles (nsec):
| 1.00th=[ 3952], 5.00th=[ 4192], 10.00th=[ 4256], 20.00th=[ 4256],
| 30.00th=[ 4320], 40.00th=[ 4320], 50.00th=[ 4384], 60.00th=[ 4384],
| 70.00th=[ 4384], 80.00th=[ 4448], 90.00th=[ 4512], 95.00th=[ 4704],
| 99.00th=[12608], 99.50th=[17536], 99.90th=[21888], 99.95th=[22400],
| 99.99th=[61184]
bw ( KiB/s): min=147349, max=148689, per=100.00%, avg=148046.36, stdev=438.09, samples=14
iops : min=294698, max=297378, avg=296092.71, stdev=876.18, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=2.12%, 10=95.87%, 20=1.74%, 50=0.24%, 100=0.02%
cpu : usr=63.53%, sys=36.44%, ctx=11, majf=0, minf=0
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2072579,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=1012MiB (1061MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-bdev_aio_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789217,
"timestamp_ms" : 1666789217545,
"time" : "Wed Oct 26 15:00:17 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 392771072,
"io_kbytes" : 383565,
"bw_bytes" : 56102138,
"bw" : 54787,
"iops" : 109574.489359,
"runtime" : 7001,
"total_ios" : 767131,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1423,
"max" : 5861,
"mean" : 1520.401970,
"stddev" : 57.282899,
"N" : 767131
},
"clat_ns" : {
"min" : 1362,
"max" : 75181,
"mean" : 7308.434213,
"stddev" : 994.829189,
"N" : 767131,
"percentile" : {
"1.000000" : 6816,
"5.000000" : 7008,
"10.000000" : 7072,
"20.000000" : 7072,
"30.000000" : 7136,
"40.000000" : 7200,
"50.000000" : 7200,
"60.000000" : 7264,
"70.000000" : 7264,
"80.000000" : 7328,
"90.000000" : 7456,
"95.000000" : 7520,
"99.000000" : 10432,
"99.500000" : 13632,
"99.900000" : 15680,
"99.950000" : 15808,
"99.990000" : 59136
}
},
"lat_ns" : {
"min" : 6832,
"max" : 76674,
"mean" : 8828.836183,
"stddev" : 997.829487,
"N" : 767131
},
"bw_min" : 54620,
"bw_max" : 55051,
"bw_agg" : 100.000000,
"bw_mean" : 54797.071429,
"bw_dev" : 124.224031,
"bw_samples" : 14,
"iops_min" : 109240,
"iops_max" : 110102,
"iops_mean" : 109594.142857,
"iops_stddev" : 248.448062,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 7.342857,
"sys_cpu" : 37.014286,
"ctx" : 767130,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 98.706740,
"20" : 1.275532,
"50" : 0.010000,
"100" : 0.014861,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=5169: Wed Oct 26 15:00:17 2022
read: IOPS=110k, BW=53.5MiB/s (56.1MB/s)(375MiB/7001msec)
slat (nsec): min=1423, max=5861, avg=1520.40, stdev=57.28
clat (nsec): min=1362, max=75181, avg=7308.43, stdev=994.83
lat (nsec): min=6832, max=76674, avg=8828.84, stdev=997.83
clat percentiles (nsec):
| 1.00th=[ 6816], 5.00th=[ 7008], 10.00th=[ 7072], 20.00th=[ 7072],
| 30.00th=[ 7136], 40.00th=[ 7200], 50.00th=[ 7200], 60.00th=[ 7264],
| 70.00th=[ 7264], 80.00th=[ 7328], 90.00th=[ 7456], 95.00th=[ 7520],
| 99.00th=[10432], 99.50th=[13632], 99.90th=[15680], 99.95th=[15808],
| 99.99th=[59136]
bw ( KiB/s): min=54620, max=55051, per=100.00%, avg=54797.07, stdev=124.22, samples=14
iops : min=109240, max=110102, avg=109594.14, stdev=248.45, samples=14
lat (usec) : 2=0.01%, 10=98.71%, 20=1.28%, 50=0.01%, 100=0.01%
cpu : usr=7.34%, sys=37.01%, ctx=767130, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=767131,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=53.5MiB/s (56.1MB/s), 53.5MiB/s-53.5MiB/s (56.1MB/s-56.1MB/s), io=375MiB (393MB), run=7001-7001msec
$ cat /tmp/fio-output_512_2_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790005,
"timestamp_ms" : 1666790005075,
"time" : "Wed Oct 26 15:13:25 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1064973824,
"io_kbytes" : 1040013,
"bw_bytes" : 152117386,
"bw" : 148552,
"iops" : 297103.985145,
"runtime" : 7001,
"total_ios" : 2080025,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1663,
"max" : 18003,
"mean" : 1897.384686,
"stddev" : 130.276176,
"N" : 2080026
},
"clat_ns" : {
"min" : 250,
"max" : 117160,
"mean" : 4608.007251,
"stddev" : 1767.883717,
"N" : 2080025,
"percentile" : {
"1.000000" : 3952,
"5.000000" : 4192,
"10.000000" : 4256,
"20.000000" : 4256,
"30.000000" : 4320,
"40.000000" : 4320,
"50.000000" : 4384,
"60.000000" : 4384,
"70.000000" : 4448,
"80.000000" : 4448,
"90.000000" : 4512,
"95.000000" : 4704,
"99.000000" : 12608,
"99.500000" : 17536,
"99.900000" : 21888,
"99.950000" : 22400,
"99.990000" : 61184
}
},
"lat_ns" : {
"min" : 5160,
"max" : 119033,
"mean" : 6505.391929,
"stddev" : 1771.826567,
"N" : 2080025
},
"bw_min" : 147892,
"bw_max" : 149767,
"bw_agg" : 100.000000,
"bw_mean" : 148627.928571,
"bw_dev" : 466.567740,
"bw_samples" : 14,
"iops_min" : 295785,
"iops_max" : 299534,
"iops_mean" : 297256.000000,
"iops_stddev" : 932.970030,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 59.700000,
"sys_cpu" : 40.271429,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 99.989808,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 2.073292,
"10" : 95.884665,
"20" : 1.778440,
"50" : 0.241055,
"100" : 0.020096,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=7997: Wed Oct 26 15:13:25 2022
read: IOPS=297k, BW=145MiB/s (152MB/s)(1016MiB/7001msec)
slat (nsec): min=1663, max=18003, avg=1897.38, stdev=130.28
clat (nsec): min=250, max=117160, avg=4608.01, stdev=1767.88
lat (usec): min=5, max=119, avg= 6.51, stdev= 1.77
clat percentiles (nsec):
| 1.00th=[ 3952], 5.00th=[ 4192], 10.00th=[ 4256], 20.00th=[ 4256],
| 30.00th=[ 4320], 40.00th=[ 4320], 50.00th=[ 4384], 60.00th=[ 4384],
| 70.00th=[ 4448], 80.00th=[ 4448], 90.00th=[ 4512], 95.00th=[ 4704],
| 99.00th=[12608], 99.50th=[17536], 99.90th=[21888], 99.95th=[22400],
| 99.99th=[61184]
bw ( KiB/s): min=147892, max=149767, per=100.00%, avg=148627.93, stdev=466.57, samples=14
iops : min=295785, max=299534, avg=297256.00, stdev=932.97, samples=14
lat (nsec) : 500=0.01%, 1000=0.01%
lat (usec) : 4=2.07%, 10=95.88%, 20=1.78%, 50=0.24%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=59.70%, sys=40.27%, ctx=10, majf=0, minf=0
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2080025,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=1016MiB (1065MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790020,
"timestamp_ms" : 1666790020382,
"time" : "Wed Oct 26 15:13:40 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1019141632,
"io_kbytes" : 995255,
"bw_bytes" : 145570865,
"bw" : 142159,
"iops" : 284317.954578,
"runtime" : 7001,
"total_ios" : 1990510,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1693,
"max" : 23505,
"mean" : 1928.243363,
"stddev" : 145.937912,
"N" : 1990510
},
"clat_ns" : {
"min" : 611,
"max" : 114164,
"mean" : 4850.434752,
"stddev" : 1749.968240,
"N" : 1990511,
"percentile" : {
"1.000000" : 4256,
"5.000000" : 4384,
"10.000000" : 4384,
"20.000000" : 4512,
"30.000000" : 4576,
"40.000000" : 4576,
"50.000000" : 4640,
"60.000000" : 4640,
"70.000000" : 4640,
"80.000000" : 4704,
"90.000000" : 4768,
"95.000000" : 4896,
"99.000000" : 12736,
"99.500000" : 18048,
"99.900000" : 21888,
"99.950000" : 22144,
"99.990000" : 60672
}
},
"lat_ns" : {
"min" : 5460,
"max" : 116057,
"mean" : 6778.678102,
"stddev" : 1753.594753,
"N" : 1990511
},
"bw_min" : 141762,
"bw_max" : 142974,
"bw_agg" : 100.000000,
"bw_mean" : 142184.785714,
"bw_dev" : 341.212809,
"bw_samples" : 14,
"iops_min" : 283524,
"iops_max" : 285949,
"iops_mean" : 284369.642857,
"iops_stddev" : 682.603569,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 56.500000,
"sys_cpu" : 43.471429,
"ctx" : 7,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 99.977795,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.017684,
"10" : 97.705714,
"20" : 1.999236,
"50" : 0.251091,
"100" : 0.019744,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=8054: Wed Oct 26 15:13:40 2022
read: IOPS=284k, BW=139MiB/s (146MB/s)(972MiB/7001msec)
slat (nsec): min=1693, max=23505, avg=1928.24, stdev=145.94
clat (nsec): min=611, max=114164, avg=4850.43, stdev=1749.97
lat (usec): min=5, max=116, avg= 6.78, stdev= 1.75
clat percentiles (nsec):
| 1.00th=[ 4256], 5.00th=[ 4384], 10.00th=[ 4384], 20.00th=[ 4512],
| 30.00th=[ 4576], 40.00th=[ 4576], 50.00th=[ 4640], 60.00th=[ 4640],
| 70.00th=[ 4640], 80.00th=[ 4704], 90.00th=[ 4768], 95.00th=[ 4896],
| 99.00th=[12736], 99.50th=[18048], 99.90th=[21888], 99.95th=[22144],
| 99.99th=[60672]
bw ( KiB/s): min=141762, max=142974, per=100.00%, avg=142184.79, stdev=341.21, samples=14
iops : min=283524, max=285949, avg=284369.64, stdev=682.60, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 4=0.02%, 10=97.71%, 20=2.00%, 50=0.25%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=56.50%, sys=43.47%, ctx=7, majf=0, minf=0
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1990510,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=139MiB/s (146MB/s), 139MiB/s-139MiB/s (146MB/s-146MB/s), io=972MiB (1019MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790035,
"timestamp_ms" : 1666790035564,
"time" : "Wed Oct 26 15:13:55 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1025831424,
"io_kbytes" : 1001788,
"bw_bytes" : 146526413,
"bw" : 143092,
"iops" : 286184.259392,
"runtime" : 7001,
"total_ios" : 2003576,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1713,
"max" : 22923,
"mean" : 1939.483181,
"stddev" : 128.999116,
"N" : 2003576
},
"clat_ns" : {
"min" : 652,
"max" : 114214,
"mean" : 4820.917562,
"stddev" : 1748.341345,
"N" : 2003577,
"percentile" : {
"1.000000" : 4256,
"5.000000" : 4384,
"10.000000" : 4448,
"20.000000" : 4512,
"30.000000" : 4512,
"40.000000" : 4576,
"50.000000" : 4576,
"60.000000" : 4576,
"70.000000" : 4640,
"80.000000" : 4640,
"90.000000" : 4768,
"95.000000" : 4896,
"99.000000" : 12736,
"99.500000" : 18048,
"99.900000" : 21888,
"99.950000" : 22144,
"99.990000" : 61184
}
},
"lat_ns" : {
"min" : 5680,
"max" : 116118,
"mean" : 6760.400710,
"stddev" : 1752.206198,
"N" : 2003577
},
"bw_min" : 142671,
"bw_max" : 143940,
"bw_agg" : 100.000000,
"bw_mean" : 143196.692308,
"bw_dev" : 456.645629,
"bw_samples" : 13,
"iops_min" : 285342,
"iops_max" : 287880,
"iops_mean" : 286393.538462,
"iops_stddev" : 913.368638,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 55.757143,
"sys_cpu" : 44.228571,
"ctx" : 9,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 99.987323,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.023558,
"10" : 97.738543,
"20" : 1.962641,
"50" : 0.251900,
"100" : 0.019815,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=8112: Wed Oct 26 15:13:55 2022
read: IOPS=286k, BW=140MiB/s (147MB/s)(978MiB/7001msec)
slat (nsec): min=1713, max=22923, avg=1939.48, stdev=129.00
clat (nsec): min=652, max=114214, avg=4820.92, stdev=1748.34
lat (usec): min=5, max=116, avg= 6.76, stdev= 1.75
clat percentiles (nsec):
| 1.00th=[ 4256], 5.00th=[ 4384], 10.00th=[ 4448], 20.00th=[ 4512],
| 30.00th=[ 4512], 40.00th=[ 4576], 50.00th=[ 4576], 60.00th=[ 4576],
| 70.00th=[ 4640], 80.00th=[ 4640], 90.00th=[ 4768], 95.00th=[ 4896],
| 99.00th=[12736], 99.50th=[18048], 99.90th=[21888], 99.95th=[22144],
| 99.99th=[61184]
bw ( KiB/s): min=142671, max=143940, per=100.00%, avg=143196.69, stdev=456.65, samples=13
iops : min=285342, max=287880, avg=286393.54, stdev=913.37, samples=13
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.02%, 10=97.74%, 20=1.96%, 50=0.25%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=55.76%, sys=44.23%, ctx=9, majf=0, minf=0
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2003576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=140MiB/s (147MB/s), 140MiB/s-140MiB/s (147MB/s-147MB/s), io=978MiB (1026MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790050,
"timestamp_ms" : 1666790050783,
"time" : "Wed Oct 26 15:14:10 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1025968128,
"io_kbytes" : 1001922,
"bw_bytes" : 146566875,
"bw" : 143131,
"iops" : 286263.142857,
"runtime" : 7000,
"total_ios" : 2003842,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1683,
"max" : 22953,
"mean" : 1917.783290,
"stddev" : 143.998960,
"N" : 2003843
},
"clat_ns" : {
"min" : 651,
"max" : 119825,
"mean" : 4827.615465,
"stddev" : 1728.532459,
"N" : 2003842,
"percentile" : {
"1.000000" : 4256,
"5.000000" : 4384,
"10.000000" : 4384,
"20.000000" : 4448,
"30.000000" : 4512,
"40.000000" : 4576,
"50.000000" : 4640,
"60.000000" : 4640,
"70.000000" : 4640,
"80.000000" : 4704,
"90.000000" : 4768,
"95.000000" : 4960,
"99.000000" : 12736,
"99.500000" : 18048,
"99.900000" : 21888,
"99.950000" : 22144,
"99.990000" : 59648
}
},
"lat_ns" : {
"min" : 5410,
"max" : 121838,
"mean" : 6745.398757,
"stddev" : 1732.651265,
"N" : 2003842
},
"bw_min" : 142876,
"bw_max" : 143492,
"bw_agg" : 100.000000,
"bw_mean" : 143137.428571,
"bw_dev" : 166.771476,
"bw_samples" : 14,
"iops_min" : 285752,
"iops_max" : 286984,
"iops_mean" : 286274.928571,
"iops_stddev" : 333.438142,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 55.185714,
"sys_cpu" : 44.785714,
"ctx" : 7,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 99.978691,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.016418,
"10" : 97.727366,
"20" : 1.985785,
"50" : 0.247275,
"100" : 0.017267,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=8172: Wed Oct 26 15:14:10 2022
read: IOPS=286k, BW=140MiB/s (147MB/s)(978MiB/7000msec)
slat (nsec): min=1683, max=22953, avg=1917.78, stdev=144.00
clat (nsec): min=651, max=119825, avg=4827.62, stdev=1728.53
lat (usec): min=5, max=121, avg= 6.75, stdev= 1.73
clat percentiles (nsec):
| 1.00th=[ 4256], 5.00th=[ 4384], 10.00th=[ 4384], 20.00th=[ 4448],
| 30.00th=[ 4512], 40.00th=[ 4576], 50.00th=[ 4640], 60.00th=[ 4640],
| 70.00th=[ 4640], 80.00th=[ 4704], 90.00th=[ 4768], 95.00th=[ 4960],
| 99.00th=[12736], 99.50th=[18048], 99.90th=[21888], 99.95th=[22144],
| 99.99th=[59648]
bw ( KiB/s): min=142876, max=143492, per=100.00%, avg=143137.43, stdev=166.77, samples=14
iops : min=285752, max=286984, avg=286274.93, stdev=333.44, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 4=0.02%, 10=97.73%, 20=1.99%, 50=0.25%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=55.19%, sys=44.79%, ctx=7, majf=0, minf=0
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2003842,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=140MiB/s (147MB/s), 140MiB/s-140MiB/s (147MB/s-147MB/s), io=978MiB (1026MB), run=7000-7000msec
$ rm /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory
$ rm /tmp/fio-output_512_2_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790064,
"timestamp_ms" : 1666790064713,
"time" : "Wed Oct 26 15:14:24 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 840821760,
"io_kbytes" : 821115,
"bw_bytes" : 120100237,
"bw" : 117285,
"iops" : 234570.489930,
"runtime" : 7001,
"total_ios" : 1642228,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1503,
"max" : 27501,
"mean" : 1736.627195,
"stddev" : 156.192241,
"N" : 1642229
},
"clat_ns" : {
"min" : 2836,
"max" : 121638,
"mean" : 6548.460939,
"stddev" : 1775.773390,
"N" : 1642229,
"percentile" : {
"1.000000" : 5920,
"5.000000" : 5920,
"10.000000" : 5984,
"20.000000" : 6048,
"30.000000" : 6112,
"40.000000" : 6176,
"50.000000" : 6240,
"60.000000" : 6304,
"70.000000" : 6368,
"80.000000" : 6432,
"90.000000" : 7136,
"95.000000" : 7584,
"99.000000" : 15040,
"99.500000" : 18560,
"99.900000" : 25984,
"99.950000" : 26496,
"99.990000" : 62208
}
},
"lat_ns" : {
"min" : 6603,
"max" : 123481,
"mean" : 8285.088133,
"stddev" : 1774.881999,
"N" : 1642229
},
"bw_min" : 116619,
"bw_max" : 117728,
"bw_agg" : 100.000000,
"bw_mean" : 117305.857143,
"bw_dev" : 299.454853,
"bw_samples" : 14,
"iops_min" : 233238,
"iops_max" : 235456,
"iops_mean" : 234611.928571,
"iops_stddev" : 598.818126,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 19.800000,
"sys_cpu" : 57.285714,
"ctx" : 824948,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.830691,
"20" : 1.792078,
"50" : 0.349099,
"100" : 0.020764,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=8219: Wed Oct 26 15:14:24 2022
read: IOPS=235k, BW=115MiB/s (120MB/s)(802MiB/7001msec)
slat (nsec): min=1503, max=27501, avg=1736.63, stdev=156.19
clat (usec): min=2, max=121, avg= 6.55, stdev= 1.78
lat (usec): min=6, max=123, avg= 8.29, stdev= 1.77
clat percentiles (nsec):
| 1.00th=[ 5920], 5.00th=[ 5920], 10.00th=[ 5984], 20.00th=[ 6048],
| 30.00th=[ 6112], 40.00th=[ 6176], 50.00th=[ 6240], 60.00th=[ 6304],
| 70.00th=[ 6368], 80.00th=[ 6432], 90.00th=[ 7136], 95.00th=[ 7584],
| 99.00th=[15040], 99.50th=[18560], 99.90th=[25984], 99.95th=[26496],
| 99.99th=[62208]
bw ( KiB/s): min=116619, max=117728, per=100.00%, avg=117305.86, stdev=299.45, samples=14
iops : min=233238, max=235456, avg=234611.93, stdev=598.82, samples=14
lat (usec) : 4=0.01%, 10=97.83%, 20=1.79%, 50=0.35%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=19.80%, sys=57.29%, ctx=824948, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1642228,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=115MiB/s (120MB/s), 115MiB/s-115MiB/s (120MB/s-120MB/s), io=802MiB (841MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790078,
"timestamp_ms" : 1666790078614,
"time" : "Wed Oct 26 15:14:38 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 837950976,
"io_kbytes" : 818311,
"bw_bytes" : 119690183,
"bw" : 116884,
"iops" : 233769.747179,
"runtime" : 7001,
"total_ios" : 1636622,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1513,
"max" : 34655,
"mean" : 1732.315724,
"stddev" : 166.086574,
"N" : 1636622
},
"clat_ns" : {
"min" : 2925,
"max" : 119975,
"mean" : 6582.195259,
"stddev" : 1784.605938,
"N" : 1636623,
"percentile" : {
"1.000000" : 5920,
"5.000000" : 5984,
"10.000000" : 5984,
"20.000000" : 6048,
"30.000000" : 6112,
"40.000000" : 6176,
"50.000000" : 6304,
"60.000000" : 6368,
"70.000000" : 6432,
"80.000000" : 6496,
"90.000000" : 7072,
"95.000000" : 7584,
"99.000000" : 15296,
"99.500000" : 18560,
"99.900000" : 25984,
"99.950000" : 26496,
"99.990000" : 62720
}
},
"lat_ns" : {
"min" : 5931,
"max" : 121779,
"mean" : 8314.510898,
"stddev" : 1783.242623,
"N" : 1636623
},
"bw_min" : 116388,
"bw_max" : 117454,
"bw_agg" : 100.000000,
"bw_mean" : 116959.071429,
"bw_dev" : 338.761878,
"bw_samples" : 14,
"iops_min" : 232776,
"iops_max" : 234908,
"iops_mean" : 233918.357143,
"iops_stddev" : 677.455262,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 20.128571,
"sys_cpu" : 56.842857,
"ctx" : 822028,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.817945,
"20" : 1.790823,
"50" : 0.360560,
"100" : 0.022241,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=8268: Wed Oct 26 15:14:38 2022
read: IOPS=234k, BW=114MiB/s (120MB/s)(799MiB/7001msec)
slat (nsec): min=1513, max=34655, avg=1732.32, stdev=166.09
clat (usec): min=2, max=119, avg= 6.58, stdev= 1.78
lat (usec): min=5, max=121, avg= 8.31, stdev= 1.78
clat percentiles (nsec):
| 1.00th=[ 5920], 5.00th=[ 5984], 10.00th=[ 5984], 20.00th=[ 6048],
| 30.00th=[ 6112], 40.00th=[ 6176], 50.00th=[ 6304], 60.00th=[ 6368],
| 70.00th=[ 6432], 80.00th=[ 6496], 90.00th=[ 7072], 95.00th=[ 7584],
| 99.00th=[15296], 99.50th=[18560], 99.90th=[25984], 99.95th=[26496],
| 99.99th=[62720]
bw ( KiB/s): min=116388, max=117454, per=100.00%, avg=116959.07, stdev=338.76, samples=14
iops : min=232776, max=234908, avg=233918.36, stdev=677.46, samples=14
lat (usec) : 4=0.01%, 10=97.82%, 20=1.79%, 50=0.36%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=20.13%, sys=56.84%, ctx=822028, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1636622,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=114MiB/s (120MB/s), 114MiB/s-114MiB/s (120MB/s-120MB/s), io=799MiB (838MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790092,
"timestamp_ms" : 1666790092370,
"time" : "Wed Oct 26 15:14:52 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 836393472,
"io_kbytes" : 816790,
"bw_bytes" : 119467714,
"bw" : 116667,
"iops" : 233335.237823,
"runtime" : 7001,
"total_ios" : 1633580,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1512,
"max" : 53531,
"mean" : 1744.740057,
"stddev" : 169.297769,
"N" : 1633580
},
"clat_ns" : {
"min" : 2875,
"max" : 119985,
"mean" : 6578.702695,
"stddev" : 1794.962820,
"N" : 1633581,
"percentile" : {
"1.000000" : 5920,
"5.000000" : 5984,
"10.000000" : 5984,
"20.000000" : 6048,
"30.000000" : 6112,
"40.000000" : 6176,
"50.000000" : 6304,
"60.000000" : 6304,
"70.000000" : 6368,
"80.000000" : 6496,
"90.000000" : 7136,
"95.000000" : 7584,
"99.000000" : 15296,
"99.500000" : 18560,
"99.900000" : 25984,
"99.950000" : 26496,
"99.990000" : 62720
}
},
"lat_ns" : {
"min" : 6793,
"max" : 121838,
"mean" : 8323.442636,
"stddev" : 1795.815444,
"N" : 1633581
},
"bw_min" : 115883,
"bw_max" : 117515,
"bw_agg" : 100.000000,
"bw_mean" : 116731.142857,
"bw_dev" : 532.190951,
"bw_samples" : 14,
"iops_min" : 231766,
"iops_max" : 235030,
"iops_mean" : 233462.428571,
"iops_stddev" : 1064.482230,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 20.028571,
"sys_cpu" : 57.300000,
"ctx" : 820575,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.820186,
"20" : 1.795872,
"50" : 0.354130,
"100" : 0.022833,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=8316: Wed Oct 26 15:14:52 2022
read: IOPS=233k, BW=114MiB/s (119MB/s)(798MiB/7001msec)
slat (nsec): min=1512, max=53531, avg=1744.74, stdev=169.30
clat (usec): min=2, max=119, avg= 6.58, stdev= 1.79
lat (usec): min=6, max=121, avg= 8.32, stdev= 1.80
clat percentiles (nsec):
| 1.00th=[ 5920], 5.00th=[ 5984], 10.00th=[ 5984], 20.00th=[ 6048],
| 30.00th=[ 6112], 40.00th=[ 6176], 50.00th=[ 6304], 60.00th=[ 6304],
| 70.00th=[ 6368], 80.00th=[ 6496], 90.00th=[ 7136], 95.00th=[ 7584],
| 99.00th=[15296], 99.50th=[18560], 99.90th=[25984], 99.95th=[26496],
| 99.99th=[62720]
bw ( KiB/s): min=115883, max=117515, per=100.00%, avg=116731.14, stdev=532.19, samples=14
iops : min=231766, max=235030, avg=233462.43, stdev=1064.48, samples=14
lat (usec) : 4=0.01%, 10=97.82%, 20=1.80%, 50=0.35%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=20.03%, sys=57.30%, ctx=820575, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1633580,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=114MiB/s (119MB/s), 114MiB/s-114MiB/s (119MB/s-119MB/s), io=798MiB (836MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790106,
"timestamp_ms" : 1666790106140,
"time" : "Wed Oct 26 15:15:06 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1580787200,
"io_kbytes" : 1543737,
"bw_bytes" : 225794486,
"bw" : 220502,
"iops" : 441004.285102,
"runtime" : 7001,
"total_ios" : 3087471,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1172,
"max" : 41017,
"mean" : 1789.167935,
"stddev" : 375.554215,
"N" : 3087472
},
"clat_ns" : {
"min" : 100,
"max" : 122480,
"mean" : 7011.723363,
"stddev" : 2863.847161,
"N" : 3087474,
"percentile" : {
"1.000000" : 4128,
"5.000000" : 4704,
"10.000000" : 4768,
"20.000000" : 6112,
"30.000000" : 6304,
"40.000000" : 6368,
"50.000000" : 6432,
"60.000000" : 6624,
"70.000000" : 6944,
"80.000000" : 7584,
"90.000000" : 8512,
"95.000000" : 9280,
"99.000000" : 20864,
"99.500000" : 25216,
"99.900000" : 27008,
"99.950000" : 33536,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 5751,
"max" : 123712,
"mean" : 8800.891246,
"stddev" : 2837.715863,
"N" : 3087474
},
"bw_min" : 219155,
"bw_max" : 221950,
"bw_agg" : 100.000000,
"bw_mean" : 220618.285714,
"bw_dev" : 970.234264,
"bw_samples" : 14,
"iops_min" : 438310,
"iops_max" : 443900,
"iops_mean" : 441236.785714,
"iops_stddev" : 1940.453523,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.271429,
"sys_cpu" : 80.428571,
"ctx" : 35740,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.016356,
"10" : 95.917889,
"20" : 2.579652,
"50" : 1.446815,
"100" : 0.039093,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=8364: Wed Oct 26 15:15:06 2022
read: IOPS=441k, BW=215MiB/s (226MB/s)(1508MiB/7001msec)
slat (nsec): min=1172, max=41017, avg=1789.17, stdev=375.55
clat (nsec): min=100, max=122480, avg=7011.72, stdev=2863.85
lat (usec): min=5, max=123, avg= 8.80, stdev= 2.84
clat percentiles (nsec):
| 1.00th=[ 4128], 5.00th=[ 4704], 10.00th=[ 4768], 20.00th=[ 6112],
| 30.00th=[ 6304], 40.00th=[ 6368], 50.00th=[ 6432], 60.00th=[ 6624],
| 70.00th=[ 6944], 80.00th=[ 7584], 90.00th=[ 8512], 95.00th=[ 9280],
| 99.00th=[20864], 99.50th=[25216], 99.90th=[27008], 99.95th=[33536],
| 99.99th=[68096]
bw ( KiB/s): min=219155, max=221950, per=100.00%, avg=220618.29, stdev=970.23, samples=14
iops : min=438310, max=443900, avg=441236.79, stdev=1940.45, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.02%, 10=95.92%, 20=2.58%, 50=1.45%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=15.27%, sys=80.43%, ctx=35740, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3087471,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=215MiB/s (226MB/s), 215MiB/s-215MiB/s (226MB/s-226MB/s), io=1508MiB (1581MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790119,
"timestamp_ms" : 1666790119919,
"time" : "Wed Oct 26 15:15:19 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1603376128,
"io_kbytes" : 1565797,
"bw_bytes" : 229021015,
"bw" : 223653,
"iops" : 447306.099129,
"runtime" : 7001,
"total_ios" : 3131590,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1162,
"max" : 24146,
"mean" : 1765.691817,
"stddev" : 356.119959,
"N" : 3131591
},
"clat_ns" : {
"min" : 90,
"max" : 125896,
"mean" : 6918.897432,
"stddev" : 2872.697041,
"N" : 3131593,
"percentile" : {
"1.000000" : 4080,
"5.000000" : 4640,
"10.000000" : 4704,
"20.000000" : 6048,
"30.000000" : 6176,
"40.000000" : 6240,
"50.000000" : 6304,
"60.000000" : 6432,
"70.000000" : 6816,
"80.000000" : 7456,
"90.000000" : 8512,
"95.000000" : 9152,
"99.000000" : 20864,
"99.500000" : 25216,
"99.900000" : 27008,
"99.950000" : 33024,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 5511,
"max" : 127119,
"mean" : 8684.589193,
"stddev" : 2830.926192,
"N" : 3131593
},
"bw_min" : 222821,
"bw_max" : 224344,
"bw_agg" : 100.000000,
"bw_mean" : 223701.928571,
"bw_dev" : 419.962354,
"bw_samples" : 14,
"iops_min" : 445642,
"iops_max" : 448688,
"iops_mean" : 447403.571429,
"iops_stddev" : 839.997868,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.257143,
"sys_cpu" : 78.285714,
"ctx" : 39299,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.170808,
"10" : 95.608844,
"20" : 2.736278,
"50" : 1.445496,
"100" : 0.037393,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=8410: Wed Oct 26 15:15:19 2022
read: IOPS=447k, BW=218MiB/s (229MB/s)(1529MiB/7001msec)
slat (nsec): min=1162, max=24146, avg=1765.69, stdev=356.12
clat (nsec): min=90, max=125896, avg=6918.90, stdev=2872.70
lat (usec): min=5, max=127, avg= 8.68, stdev= 2.83
clat percentiles (nsec):
| 1.00th=[ 4080], 5.00th=[ 4640], 10.00th=[ 4704], 20.00th=[ 6048],
| 30.00th=[ 6176], 40.00th=[ 6240], 50.00th=[ 6304], 60.00th=[ 6432],
| 70.00th=[ 6816], 80.00th=[ 7456], 90.00th=[ 8512], 95.00th=[ 9152],
| 99.00th=[20864], 99.50th=[25216], 99.90th=[27008], 99.95th=[33024],
| 99.99th=[68096]
bw ( KiB/s): min=222821, max=224344, per=100.00%, avg=223701.93, stdev=419.96, samples=14
iops : min=445642, max=448688, avg=447403.57, stdev=840.00, samples=14
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.17%, 10=95.61%, 20=2.74%, 50=1.45%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=17.26%, sys=78.29%, ctx=39299, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3131590,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=218MiB/s (229MB/s), 218MiB/s-218MiB/s (229MB/s-229MB/s), io=1529MiB (1603MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790133,
"timestamp_ms" : 1666790133702,
"time" : "Wed Oct 26 15:15:33 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1593121280,
"io_kbytes" : 1555782,
"bw_bytes" : 227556246,
"bw" : 222222,
"iops" : 444445.222111,
"runtime" : 7001,
"total_ios" : 3111561,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1162,
"max" : 18786,
"mean" : 1779.421280,
"stddev" : 358.143700,
"N" : 3111562
},
"clat_ns" : {
"min" : 90,
"max" : 126347,
"mean" : 6962.273469,
"stddev" : 2892.549055,
"N" : 3111564,
"percentile" : {
"1.000000" : 4128,
"5.000000" : 4640,
"10.000000" : 4768,
"20.000000" : 6112,
"30.000000" : 6176,
"40.000000" : 6304,
"50.000000" : 6368,
"60.000000" : 6496,
"70.000000" : 6880,
"80.000000" : 7456,
"90.000000" : 8512,
"95.000000" : 9152,
"99.000000" : 20864,
"99.500000" : 25472,
"99.900000" : 27264,
"99.950000" : 34048,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 5530,
"max" : 127549,
"mean" : 8741.694742,
"stddev" : 2854.750029,
"N" : 3111564
},
"bw_min" : 221009,
"bw_max" : 223129,
"bw_agg" : 100.000000,
"bw_mean" : 222349.285714,
"bw_dev" : 743.621536,
"bw_samples" : 14,
"iops_min" : 442018,
"iops_max" : 446258,
"iops_mean" : 444698.428571,
"iops_stddev" : 1487.477820,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 16.157143,
"sys_cpu" : 79.471429,
"ctx" : 36179,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.077582,
"10" : 95.868408,
"20" : 2.476474,
"50" : 1.537267,
"100" : 0.039498,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=8456: Wed Oct 26 15:15:33 2022
read: IOPS=444k, BW=217MiB/s (228MB/s)(1519MiB/7001msec)
slat (nsec): min=1162, max=18786, avg=1779.42, stdev=358.14
clat (nsec): min=90, max=126347, avg=6962.27, stdev=2892.55
lat (usec): min=5, max=127, avg= 8.74, stdev= 2.85
clat percentiles (nsec):
| 1.00th=[ 4128], 5.00th=[ 4640], 10.00th=[ 4768], 20.00th=[ 6112],
| 30.00th=[ 6176], 40.00th=[ 6304], 50.00th=[ 6368], 60.00th=[ 6496],
| 70.00th=[ 6880], 80.00th=[ 7456], 90.00th=[ 8512], 95.00th=[ 9152],
| 99.00th=[20864], 99.50th=[25472], 99.90th=[27264], 99.95th=[34048],
| 99.99th=[68096]
bw ( KiB/s): min=221009, max=223129, per=100.00%, avg=222349.29, stdev=743.62, samples=14
iops : min=442018, max=446258, avg=444698.43, stdev=1487.48, samples=14
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.08%, 10=95.87%, 20=2.48%, 50=1.54%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=16.16%, sys=79.47%, ctx=36179, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3111561,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=217MiB/s (228MB/s), 217MiB/s-217MiB/s (228MB/s-228MB/s), io=1519MiB (1593MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789232,
"timestamp_ms" : 1666789232866,
"time" : "Wed Oct 26 15:00:32 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 584781824,
"io_kbytes" : 571076,
"bw_bytes" : 83528327,
"bw" : 81570,
"iops" : 163141.265533,
"runtime" : 7001,
"total_ios" : 1142152,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 230,
"max" : 3617,
"mean" : 269.127672,
"stddev" : 21.625019,
"N" : 1142152
},
"clat_ns" : {
"min" : 2775,
"max" : 111990,
"mean" : 5644.366970,
"stddev" : 1128.740165,
"N" : 1142152,
"percentile" : {
"1.000000" : 5344,
"5.000000" : 5344,
"10.000000" : 5408,
"20.000000" : 5408,
"30.000000" : 5472,
"40.000000" : 5472,
"50.000000" : 5472,
"60.000000" : 5536,
"70.000000" : 5536,
"80.000000" : 5600,
"90.000000" : 5664,
"95.000000" : 5728,
"99.000000" : 11456,
"99.500000" : 12480,
"99.900000" : 16768,
"99.950000" : 17024,
"99.990000" : 58624
}
},
"lat_ns" : {
"min" : 5560,
"max" : 112270,
"mean" : 5913.494643,
"stddev" : 1129.505716,
"N" : 1142152
},
"bw_min" : 81452,
"bw_max" : 81832,
"bw_agg" : 100.000000,
"bw_mean" : 81605.142857,
"bw_dev" : 116.354139,
"bw_samples" : 14,
"iops_min" : 162905,
"iops_max" : 163665,
"iops_mean" : 163210.428571,
"iops_stddev" : 232.757290,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 13,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 98.485491,
"20" : 1.498575,
"50" : 0.010000,
"100" : 0.013221,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=5227: Wed Oct 26 15:00:32 2022
read: IOPS=163k, BW=79.7MiB/s (83.5MB/s)(558MiB/7001msec)
slat (nsec): min=230, max=3617, avg=269.13, stdev=21.63
clat (usec): min=2, max=111, avg= 5.64, stdev= 1.13
lat (usec): min=5, max=112, avg= 5.91, stdev= 1.13
clat percentiles (nsec):
| 1.00th=[ 5344], 5.00th=[ 5344], 10.00th=[ 5408], 20.00th=[ 5408],
| 30.00th=[ 5472], 40.00th=[ 5472], 50.00th=[ 5472], 60.00th=[ 5536],
| 70.00th=[ 5536], 80.00th=[ 5600], 90.00th=[ 5664], 95.00th=[ 5728],
| 99.00th=[11456], 99.50th=[12480], 99.90th=[16768], 99.95th=[17024],
| 99.99th=[58624]
bw ( KiB/s): min=81452, max=81832, per=100.00%, avg=81605.14, stdev=116.35, samples=14
iops : min=162905, max=163665, avg=163210.43, stdev=232.76, samples=14
lat (usec) : 4=0.01%, 10=98.49%, 20=1.50%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=13, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1142152,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=79.7MiB/s (83.5MB/s), 79.7MiB/s-79.7MiB/s (83.5MB/s-83.5MB/s), io=558MiB (585MB), run=7001-7001msec
$ cat /tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790147,
"timestamp_ms" : 1666790147445,
"time" : "Wed Oct 26 15:15:47 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1479533568,
"io_kbytes" : 1444857,
"bw_bytes" : 211331748,
"bw" : 206378,
"iops" : 412756.891873,
"runtime" : 7001,
"total_ios" : 2889711,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1232,
"max" : 22802,
"mean" : 1856.760263,
"stddev" : 376.862601,
"N" : 2889711
},
"clat_ns" : {
"min" : 140,
"max" : 125245,
"mean" : 7527.013691,
"stddev" : 2833.703087,
"N" : 2889714,
"percentile" : {
"1.000000" : 4320,
"5.000000" : 4896,
"10.000000" : 5792,
"20.000000" : 6624,
"30.000000" : 6688,
"40.000000" : 7136,
"50.000000" : 7264,
"60.000000" : 7328,
"70.000000" : 7520,
"80.000000" : 7904,
"90.000000" : 8512,
"95.000000" : 9408,
"99.000000" : 20864,
"99.500000" : 25216,
"99.900000" : 27520,
"99.950000" : 34048,
"99.990000" : 69120
}
},
"lat_ns" : {
"min" : 5701,
"max" : 127048,
"mean" : 9383.773646,
"stddev" : 2814.443814,
"N" : 2889714
},
"bw_min" : 205006,
"bw_max" : 208277,
"bw_agg" : 100.000000,
"bw_mean" : 206571.076923,
"bw_dev" : 1108.539088,
"bw_samples" : 13,
"iops_min" : 410012,
"iops_max" : 416554,
"iops_mean" : 413142.153846,
"iops_stddev" : 2216.807195,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 19.342857,
"sys_cpu" : 76.642857,
"ctx" : 35152,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 95.738432,
"20" : 2.739409,
"50" : 1.478556,
"100" : 0.041319,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=8503: Wed Oct 26 15:15:47 2022
read: IOPS=413k, BW=202MiB/s (211MB/s)(1411MiB/7001msec)
slat (nsec): min=1232, max=22802, avg=1856.76, stdev=376.86
clat (nsec): min=140, max=125245, avg=7527.01, stdev=2833.70
lat (usec): min=5, max=127, avg= 9.38, stdev= 2.81
clat percentiles (nsec):
| 1.00th=[ 4320], 5.00th=[ 4896], 10.00th=[ 5792], 20.00th=[ 6624],
| 30.00th=[ 6688], 40.00th=[ 7136], 50.00th=[ 7264], 60.00th=[ 7328],
| 70.00th=[ 7520], 80.00th=[ 7904], 90.00th=[ 8512], 95.00th=[ 9408],
| 99.00th=[20864], 99.50th=[25216], 99.90th=[27520], 99.95th=[34048],
| 99.99th=[69120]
bw ( KiB/s): min=205006, max=208277, per=100.00%, avg=206571.08, stdev=1108.54, samples=13
iops : min=410012, max=416554, avg=413142.15, stdev=2216.81, samples=13
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=95.74%, 20=2.74%, 50=1.48%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=19.34%, sys=76.64%, ctx=35152, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2889711,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=202MiB/s (211MB/s), 202MiB/s-202MiB/s (211MB/s-211MB/s), io=1411MiB (1480MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790161,
"timestamp_ms" : 1666790161185,
"time" : "Wed Oct 26 15:16:01 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1513281024,
"io_kbytes" : 1477813,
"bw_bytes" : 216152124,
"bw" : 211086,
"iops" : 422171.546922,
"runtime" : 7001,
"total_ios" : 2955623,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1212,
"max" : 68398,
"mean" : 1804.931661,
"stddev" : 383.353388,
"N" : 2955624
},
"clat_ns" : {
"min" : 130,
"max" : 121648,
"mean" : 7361.880692,
"stddev" : 2801.249674,
"N" : 2955626,
"percentile" : {
"1.000000" : 4256,
"5.000000" : 4832,
"10.000000" : 5088,
"20.000000" : 6496,
"30.000000" : 6560,
"40.000000" : 6688,
"50.000000" : 7072,
"60.000000" : 7200,
"70.000000" : 7392,
"80.000000" : 7840,
"90.000000" : 8384,
"95.000000" : 9280,
"99.000000" : 20864,
"99.500000" : 24960,
"99.900000" : 27264,
"99.950000" : 33024,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 5770,
"max" : 122910,
"mean" : 9166.812258,
"stddev" : 2786.079322,
"N" : 2955626
},
"bw_min" : 209124,
"bw_max" : 212490,
"bw_agg" : 100.000000,
"bw_mean" : 211123.571429,
"bw_dev" : 1103.367274,
"bw_samples" : 14,
"iops_min" : 418248,
"iops_max" : 424980,
"iops_mean" : 422247.142857,
"iops_stddev" : 2206.773449,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 19.828571,
"sys_cpu" : 76.085714,
"ctx" : 35312,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 95.890816,
"20" : 2.646379,
"50" : 1.422881,
"100" : 0.037623,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=8549: Wed Oct 26 15:16:01 2022
read: IOPS=422k, BW=206MiB/s (216MB/s)(1443MiB/7001msec)
slat (nsec): min=1212, max=68398, avg=1804.93, stdev=383.35
clat (nsec): min=130, max=121648, avg=7361.88, stdev=2801.25
lat (usec): min=5, max=122, avg= 9.17, stdev= 2.79
clat percentiles (nsec):
| 1.00th=[ 4256], 5.00th=[ 4832], 10.00th=[ 5088], 20.00th=[ 6496],
| 30.00th=[ 6560], 40.00th=[ 6688], 50.00th=[ 7072], 60.00th=[ 7200],
| 70.00th=[ 7392], 80.00th=[ 7840], 90.00th=[ 8384], 95.00th=[ 9280],
| 99.00th=[20864], 99.50th=[24960], 99.90th=[27264], 99.95th=[33024],
| 99.99th=[68096]
bw ( KiB/s): min=209124, max=212490, per=100.00%, avg=211123.57, stdev=1103.37, samples=14
iops : min=418248, max=424980, avg=422247.14, stdev=2206.77, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=95.89%, 20=2.65%, 50=1.42%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=19.83%, sys=76.09%, ctx=35312, majf=0, minf=1
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2955623,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=206MiB/s (216MB/s), 206MiB/s-206MiB/s (216MB/s-216MB/s), io=1443MiB (1513MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790174,
"timestamp_ms" : 1666790174990,
"time" : "Wed Oct 26 15:16:14 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1493146112,
"io_kbytes" : 1458150,
"bw_bytes" : 213276119,
"bw" : 208277,
"iops" : 416554.349379,
"runtime" : 7001,
"total_ios" : 2916297,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1222,
"max" : 23835,
"mean" : 1835.246559,
"stddev" : 380.983468,
"N" : 2916298
},
"clat_ns" : {
"min" : 140,
"max" : 117631,
"mean" : 7459.801219,
"stddev" : 2783.673537,
"N" : 2916300,
"percentile" : {
"1.000000" : 4256,
"5.000000" : 4896,
"10.000000" : 5792,
"20.000000" : 6560,
"30.000000" : 6624,
"40.000000" : 6816,
"50.000000" : 7200,
"60.000000" : 7264,
"70.000000" : 7520,
"80.000000" : 7904,
"90.000000" : 8384,
"95.000000" : 9408,
"99.000000" : 20864,
"99.500000" : 25216,
"99.900000" : 27264,
"99.950000" : 33024,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 5740,
"max" : 119665,
"mean" : 9295.047879,
"stddev" : 2765.898562,
"N" : 2916300
},
"bw_min" : 206224,
"bw_max" : 211144,
"bw_agg" : 100.000000,
"bw_mean" : 208366.357143,
"bw_dev" : 1490.058678,
"bw_samples" : 14,
"iops_min" : 412448,
"iops_max" : 422288,
"iops_mean" : 416732.642857,
"iops_stddev" : 2980.303175,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 20.971429,
"sys_cpu" : 75.014286,
"ctx" : 35185,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 95.807011,
"20" : 2.722768,
"50" : 1.432056,
"100" : 0.036450,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=8596: Wed Oct 26 15:16:14 2022
read: IOPS=417k, BW=203MiB/s (213MB/s)(1424MiB/7001msec)
slat (nsec): min=1222, max=23835, avg=1835.25, stdev=380.98
clat (nsec): min=140, max=117631, avg=7459.80, stdev=2783.67
lat (usec): min=5, max=119, avg= 9.30, stdev= 2.77
clat percentiles (nsec):
| 1.00th=[ 4256], 5.00th=[ 4896], 10.00th=[ 5792], 20.00th=[ 6560],
| 30.00th=[ 6624], 40.00th=[ 6816], 50.00th=[ 7200], 60.00th=[ 7264],
| 70.00th=[ 7520], 80.00th=[ 7904], 90.00th=[ 8384], 95.00th=[ 9408],
| 99.00th=[20864], 99.50th=[25216], 99.90th=[27264], 99.95th=[33024],
| 99.99th=[68096]
bw ( KiB/s): min=206224, max=211144, per=100.00%, avg=208366.36, stdev=1490.06, samples=14
iops : min=412448, max=422288, avg=416732.64, stdev=2980.30, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=95.81%, 20=2.72%, 50=1.43%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=20.97%, sys=75.01%, ctx=35185, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2916297,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=203MiB/s (213MB/s), 203MiB/s-203MiB/s (213MB/s-213MB/s), io=1424MiB (1493MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790190,
"timestamp_ms" : 1666790190135,
"time" : "Wed Oct 26 15:16:30 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1345384448,
"io_kbytes" : 1313852,
"bw_bytes" : 192170325,
"bw" : 187666,
"iops" : 375332.238252,
"runtime" : 7001,
"total_ios" : 2627701,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 651,
"max" : 13565,
"mean" : 938.106651,
"stddev" : 69.869104,
"N" : 2627701
},
"clat_ns" : {
"min" : 1402,
"max" : 108673,
"mean" : 9369.322797,
"stddev" : 2206.289705,
"N" : 2627704,
"percentile" : {
"1.000000" : 7712,
"5.000000" : 8256,
"10.000000" : 8512,
"20.000000" : 8640,
"30.000000" : 8896,
"40.000000" : 8896,
"50.000000" : 9024,
"60.000000" : 9152,
"70.000000" : 9280,
"80.000000" : 9408,
"90.000000" : 9664,
"95.000000" : 10688,
"99.000000" : 20096,
"99.500000" : 24192,
"99.900000" : 29056,
"99.950000" : 34048,
"99.990000" : 67072
}
},
"lat_ns" : {
"min" : 7174,
"max" : 109625,
"mean" : 10307.429430,
"stddev" : 2207.091785,
"N" : 2627704
},
"bw_min" : 187307,
"bw_max" : 188201,
"bw_agg" : 100.000000,
"bw_mean" : 187705.857143,
"bw_dev" : 270.564327,
"bw_samples" : 14,
"iops_min" : 374614,
"iops_max" : 376402,
"iops_mean" : 375411.785714,
"iops_stddev" : 541.060523,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 12,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 3.338127,
"4" : 96.661683,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 93.003199,
"20" : 5.969895,
"50" : 0.989154,
"100" : 0.037714,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=8655: Wed Oct 26 15:16:30 2022
read: IOPS=375k, BW=183MiB/s (192MB/s)(1283MiB/7001msec)
slat (nsec): min=651, max=13565, avg=938.11, stdev=69.87
clat (nsec): min=1402, max=108673, avg=9369.32, stdev=2206.29
lat (usec): min=7, max=109, avg=10.31, stdev= 2.21
clat percentiles (nsec):
| 1.00th=[ 7712], 5.00th=[ 8256], 10.00th=[ 8512], 20.00th=[ 8640],
| 30.00th=[ 8896], 40.00th=[ 8896], 50.00th=[ 9024], 60.00th=[ 9152],
| 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[ 9664], 95.00th=[10688],
| 99.00th=[20096], 99.50th=[24192], 99.90th=[29056], 99.95th=[34048],
| 99.99th=[67072]
bw ( KiB/s): min=187307, max=188201, per=100.00%, avg=187705.86, stdev=270.56, samples=14
iops : min=374614, max=376402, avg=375411.79, stdev=541.06, samples=14
lat (usec) : 2=0.01%, 10=93.00%, 20=5.97%, 50=0.99%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=12, majf=0, minf=0
IO depths : 1=0.1%, 2=3.3%, 4=96.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2627701,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=183MiB/s (192MB/s), 183MiB/s-183MiB/s (192MB/s-192MB/s), io=1283MiB (1345MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory
$ rm /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790205,
"timestamp_ms" : 1666790205340,
"time" : "Wed Oct 26 15:16:45 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1416632832,
"io_kbytes" : 1383430,
"bw_bytes" : 202347212,
"bw" : 197604,
"iops" : 395208.970147,
"runtime" : 7001,
"total_ios" : 2766858,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 171,
"max" : 16481,
"mean" : 269.504734,
"stddev" : 26.227924,
"N" : 2766858
},
"clat_ns" : {
"min" : 380,
"max" : 121328,
"mean" : 9537.189593,
"stddev" : 2328.868613,
"N" : 2766861,
"percentile" : {
"1.000000" : 6432,
"5.000000" : 7392,
"10.000000" : 8032,
"20.000000" : 8768,
"30.000000" : 9024,
"40.000000" : 9152,
"50.000000" : 9280,
"60.000000" : 9408,
"70.000000" : 9536,
"80.000000" : 9664,
"90.000000" : 10304,
"95.000000" : 11840,
"99.000000" : 20096,
"99.500000" : 21888,
"99.900000" : 27264,
"99.950000" : 33024,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 5581,
"max" : 121598,
"mean" : 9806.694310,
"stddev" : 2330.569751,
"N" : 2766861
},
"bw_min" : 197016,
"bw_max" : 198564,
"bw_agg" : 100.000000,
"bw_mean" : 197684.571429,
"bw_dev" : 480.434213,
"bw_samples" : 14,
"iops_min" : 394032,
"iops_max" : 397128,
"iops_mean" : 395369.142857,
"iops_stddev" : 960.868426,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 14,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 12.759744,
"4" : 87.237184,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 87.316841,
"20" : 11.658495,
"50" : 0.982631,
"100" : 0.041672,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=8715: Wed Oct 26 15:16:45 2022
read: IOPS=395k, BW=193MiB/s (202MB/s)(1351MiB/7001msec)
slat (nsec): min=171, max=16481, avg=269.50, stdev=26.23
clat (nsec): min=380, max=121328, avg=9537.19, stdev=2328.87
lat (usec): min=5, max=121, avg= 9.81, stdev= 2.33
clat percentiles (nsec):
| 1.00th=[ 6432], 5.00th=[ 7392], 10.00th=[ 8032], 20.00th=[ 8768],
| 30.00th=[ 9024], 40.00th=[ 9152], 50.00th=[ 9280], 60.00th=[ 9408],
| 70.00th=[ 9536], 80.00th=[ 9664], 90.00th=[10304], 95.00th=[11840],
| 99.00th=[20096], 99.50th=[21888], 99.90th=[27264], 99.95th=[33024],
| 99.99th=[68096]
bw ( KiB/s): min=197016, max=198564, per=100.00%, avg=197684.57, stdev=480.43, samples=14
iops : min=394032, max=397128, avg=395369.14, stdev=960.87, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 10=87.32%, 20=11.66%, 50=0.98%, 100=0.04%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=14, majf=0, minf=0
IO depths : 1=0.1%, 2=12.8%, 4=87.2%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2766858,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=193MiB/s (202MB/s), 193MiB/s-193MiB/s (202MB/s-202MB/s), io=1351MiB (1417MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790220,
"timestamp_ms" : 1666790220567,
"time" : "Wed Oct 26 15:17:00 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1625583104,
"io_kbytes" : 1587483,
"bw_bytes" : 232192987,
"bw" : 226750,
"iops" : 453501.499786,
"runtime" : 7001,
"total_ios" : 3174964,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 200,
"max" : 19697,
"mean" : 283.869127,
"stddev" : 75.463763,
"N" : 3174964
},
"clat_ns" : {
"min" : 501,
"max" : 111770,
"mean" : 8226.494525,
"stddev" : 2228.476383,
"N" : 3174967,
"percentile" : {
"1.000000" : 6368,
"5.000000" : 6752,
"10.000000" : 7008,
"20.000000" : 7456,
"30.000000" : 7584,
"40.000000" : 7648,
"50.000000" : 7776,
"60.000000" : 7904,
"70.000000" : 8096,
"80.000000" : 8512,
"90.000000" : 9280,
"95.000000" : 10176,
"99.000000" : 18816,
"99.500000" : 21376,
"99.900000" : 27520,
"99.950000" : 33536,
"99.990000" : 67072
}
},
"lat_ns" : {
"min" : 5861,
"max" : 112050,
"mean" : 8510.363694,
"stddev" : 2229.327804,
"N" : 3174967
},
"bw_min" : 226564,
"bw_max" : 227133,
"bw_agg" : 100.000000,
"bw_mean" : 226841.785714,
"bw_dev" : 143.797496,
"bw_samples" : 14,
"iops_min" : 453128,
"iops_max" : 454266,
"iops_mean" : 453683.642857,
"iops_stddev" : 287.590416,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 1.998952,
"4" : 97.991001,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 94.702460,
"20" : 4.585154,
"50" : 0.674307,
"100" : 0.037197,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=8773: Wed Oct 26 15:17:00 2022
read: IOPS=454k, BW=221MiB/s (232MB/s)(1550MiB/7001msec)
slat (nsec): min=200, max=19697, avg=283.87, stdev=75.46
clat (nsec): min=501, max=111770, avg=8226.49, stdev=2228.48
lat (usec): min=5, max=112, avg= 8.51, stdev= 2.23
clat percentiles (nsec):
| 1.00th=[ 6368], 5.00th=[ 6752], 10.00th=[ 7008], 20.00th=[ 7456],
| 30.00th=[ 7584], 40.00th=[ 7648], 50.00th=[ 7776], 60.00th=[ 7904],
| 70.00th=[ 8096], 80.00th=[ 8512], 90.00th=[ 9280], 95.00th=[10176],
| 99.00th=[18816], 99.50th=[21376], 99.90th=[27520], 99.95th=[33536],
| 99.99th=[67072]
bw ( KiB/s): min=226564, max=227133, per=100.00%, avg=226841.79, stdev=143.80, samples=14
iops : min=453128, max=454266, avg=453683.64, stdev=287.59, samples=14
lat (nsec) : 750=0.01%
lat (usec) : 10=94.70%, 20=4.59%, 50=0.67%, 100=0.04%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=10, majf=0, minf=0
IO depths : 1=0.1%, 2=2.0%, 4=98.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3174964,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=221MiB/s (232MB/s), 221MiB/s-221MiB/s (232MB/s-232MB/s), io=1550MiB (1626MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790234,
"timestamp_ms" : 1666790234399,
"time" : "Wed Oct 26 15:17:14 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1530035200,
"io_kbytes" : 1494175,
"bw_bytes" : 218545236,
"bw" : 213423,
"iops" : 426845.593487,
"runtime" : 7001,
"total_ios" : 2988346,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1313,
"max" : 23915,
"mean" : 1844.950892,
"stddev" : 353.846891,
"N" : 2988347
},
"clat_ns" : {
"min" : 110,
"max" : 126256,
"mean" : 7246.118301,
"stddev" : 2909.869603,
"N" : 2988349,
"percentile" : {
"1.000000" : 4192,
"5.000000" : 4320,
"10.000000" : 4768,
"20.000000" : 6240,
"30.000000" : 6496,
"40.000000" : 6560,
"50.000000" : 6752,
"60.000000" : 7072,
"70.000000" : 7520,
"80.000000" : 8256,
"90.000000" : 8768,
"95.000000" : 9408,
"99.000000" : 20608,
"99.500000" : 24960,
"99.900000" : 27008,
"99.950000" : 33024,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 5380,
"max" : 127619,
"mean" : 9091.069296,
"stddev" : 2889.364715,
"N" : 2988349
},
"bw_min" : 212813,
"bw_max" : 214359,
"bw_agg" : 100.000000,
"bw_mean" : 213525.857143,
"bw_dev" : 374.802688,
"bw_samples" : 14,
"iops_min" : 425627,
"iops_max" : 428718,
"iops_mean" : 427051.928571,
"iops_stddev" : 749.479559,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.014286,
"sys_cpu" : 78.814286,
"ctx" : 36626,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 95.847569,
"20" : 2.632359,
"50" : 1.478845,
"100" : 0.039487,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 4302124,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 23026,
"write_ticks" : 0,
"in_queue" : 23026,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=8823: Wed Oct 26 15:17:14 2022
read: IOPS=427k, BW=208MiB/s (219MB/s)(1459MiB/7001msec)
slat (nsec): min=1313, max=23915, avg=1844.95, stdev=353.85
clat (nsec): min=110, max=126256, avg=7246.12, stdev=2909.87
lat (usec): min=5, max=127, avg= 9.09, stdev= 2.89
clat percentiles (nsec):
| 1.00th=[ 4192], 5.00th=[ 4320], 10.00th=[ 4768], 20.00th=[ 6240],
| 30.00th=[ 6496], 40.00th=[ 6560], 50.00th=[ 6752], 60.00th=[ 7072],
| 70.00th=[ 7520], 80.00th=[ 8256], 90.00th=[ 8768], 95.00th=[ 9408],
| 99.00th=[20608], 99.50th=[24960], 99.90th=[27008], 99.95th=[33024],
| 99.99th=[68096]
bw ( KiB/s): min=212813, max=214359, per=100.00%, avg=213525.86, stdev=374.80, samples=14
iops : min=425627, max=428718, avg=427051.93, stdev=749.48, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=95.85%, 20=2.63%, 50=1.48%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=17.01%, sys=78.81%, ctx=36626, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2988346,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=208MiB/s (219MB/s), 208MiB/s-208MiB/s (219MB/s-219MB/s), io=1459MiB (1530MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=4302124/0, merge=0/0, ticks=23026/0, in_queue=23026, util=99.09%
$ rm /tmp/fio-output_512_4_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790248,
"timestamp_ms" : 1666790248087,
"time" : "Wed Oct 26 15:17:28 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1519099904,
"io_kbytes" : 1483496,
"bw_bytes" : 216983274,
"bw" : 211897,
"iops" : 423794.886445,
"runtime" : 7001,
"total_ios" : 2966988,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1322,
"max" : 22562,
"mean" : 1861.858082,
"stddev" : 352.891983,
"N" : 2966989
},
"clat_ns" : {
"min" : 100,
"max" : 122830,
"mean" : 7297.526278,
"stddev" : 2887.510987,
"N" : 2966991,
"percentile" : {
"1.000000" : 4192,
"5.000000" : 4320,
"10.000000" : 4832,
"20.000000" : 6368,
"30.000000" : 6560,
"40.000000" : 6624,
"50.000000" : 6880,
"60.000000" : 7200,
"70.000000" : 7520,
"80.000000" : 8256,
"90.000000" : 8768,
"95.000000" : 9408,
"99.000000" : 20608,
"99.500000" : 24960,
"99.900000" : 27008,
"99.950000" : 33024,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 5731,
"max" : 124212,
"mean" : 9159.384584,
"stddev" : 2868.348094,
"N" : 2966991
},
"bw_min" : 211271,
"bw_max" : 212946,
"bw_agg" : 100.000000,
"bw_mean" : 212027.500000,
"bw_dev" : 425.605225,
"bw_samples" : 14,
"iops_min" : 422542,
"iops_max" : 425892,
"iops_mean" : 424055.357143,
"iops_stddev" : 851.333308,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.200000,
"sys_cpu" : 78.642857,
"ctx" : 35976,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 95.813296,
"20" : 2.761386,
"50" : 1.384872,
"100" : 0.039299,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 4273785,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 22940,
"write_ticks" : 0,
"in_queue" : 22940,
"util" : 99.098564
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=8869: Wed Oct 26 15:17:28 2022
read: IOPS=424k, BW=207MiB/s (217MB/s)(1449MiB/7001msec)
slat (nsec): min=1322, max=22562, avg=1861.86, stdev=352.89
clat (nsec): min=100, max=122830, avg=7297.53, stdev=2887.51
lat (usec): min=5, max=124, avg= 9.16, stdev= 2.87
clat percentiles (nsec):
| 1.00th=[ 4192], 5.00th=[ 4320], 10.00th=[ 4832], 20.00th=[ 6368],
| 30.00th=[ 6560], 40.00th=[ 6624], 50.00th=[ 6880], 60.00th=[ 7200],
| 70.00th=[ 7520], 80.00th=[ 8256], 90.00th=[ 8768], 95.00th=[ 9408],
| 99.00th=[20608], 99.50th=[24960], 99.90th=[27008], 99.95th=[33024],
| 99.99th=[68096]
bw ( KiB/s): min=211271, max=212946, per=100.00%, avg=212027.50, stdev=425.61, samples=14
iops : min=422542, max=425892, avg=424055.36, stdev=851.33, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=95.81%, 20=2.76%, 50=1.38%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=17.20%, sys=78.64%, ctx=35976, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2966988,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=1449MiB (1519MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=4273785/0, merge=0/0, ticks=22940/0, in_queue=22940, util=99.10%
$ rm /tmp/fio-output_512_4_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790261,
"timestamp_ms" : 1666790261828,
"time" : "Wed Oct 26 15:17:41 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1502555648,
"io_kbytes" : 1467339,
"bw_bytes" : 214620146,
"bw" : 209589,
"iops" : 419179.402942,
"runtime" : 7001,
"total_ios" : 2934675,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1333,
"max" : 17603,
"mean" : 1865.526482,
"stddev" : 355.622282,
"N" : 2934676
},
"clat_ns" : {
"min" : 100,
"max" : 118191,
"mean" : 7382.369814,
"stddev" : 2876.847571,
"N" : 2934678,
"percentile" : {
"1.000000" : 4256,
"5.000000" : 4320,
"10.000000" : 4896,
"20.000000" : 6496,
"30.000000" : 6624,
"40.000000" : 6752,
"50.000000" : 7136,
"60.000000" : 7264,
"70.000000" : 7648,
"80.000000" : 8256,
"90.000000" : 8640,
"95.000000" : 9408,
"99.000000" : 20864,
"99.500000" : 25216,
"99.900000" : 27008,
"99.950000" : 33024,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 5760,
"max" : 119564,
"mean" : 9247.896274,
"stddev" : 2865.318682,
"N" : 2934678
},
"bw_min" : 208949,
"bw_max" : 210632,
"bw_agg" : 100.000000,
"bw_mean" : 209732.785714,
"bw_dev" : 576.971694,
"bw_samples" : 14,
"iops_min" : 417898,
"iops_max" : 421264,
"iops_mean" : 419465.785714,
"iops_stddev" : 1153.870889,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.242857,
"sys_cpu" : 78.757143,
"ctx" : 32783,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 95.971786,
"20" : 2.527912,
"50" : 1.462308,
"100" : 0.035949,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 4230620,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 22672,
"write_ticks" : 0,
"in_queue" : 22672,
"util" : 99.098564
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=8917: Wed Oct 26 15:17:41 2022
read: IOPS=419k, BW=205MiB/s (215MB/s)(1433MiB/7001msec)
slat (nsec): min=1333, max=17603, avg=1865.53, stdev=355.62
clat (nsec): min=100, max=118191, avg=7382.37, stdev=2876.85
lat (usec): min=5, max=119, avg= 9.25, stdev= 2.87
clat percentiles (nsec):
| 1.00th=[ 4256], 5.00th=[ 4320], 10.00th=[ 4896], 20.00th=[ 6496],
| 30.00th=[ 6624], 40.00th=[ 6752], 50.00th=[ 7136], 60.00th=[ 7264],
| 70.00th=[ 7648], 80.00th=[ 8256], 90.00th=[ 8640], 95.00th=[ 9408],
| 99.00th=[20864], 99.50th=[25216], 99.90th=[27008], 99.95th=[33024],
| 99.99th=[68096]
bw ( KiB/s): min=208949, max=210632, per=100.00%, avg=209732.79, stdev=576.97, samples=14
iops : min=417898, max=421264, avg=419465.79, stdev=1153.87, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=95.97%, 20=2.53%, 50=1.46%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=17.24%, sys=78.76%, ctx=32783, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2934675,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=205MiB/s (215MB/s), 205MiB/s-205MiB/s (215MB/s-215MB/s), io=1433MiB (1503MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=4230620/0, merge=0/0, ticks=22672/0, in_queue=22672, util=99.10%
$ rm /tmp/fio-output_512_4_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790276,
"timestamp_ms" : 1666790276804,
"time" : "Wed Oct 26 15:17:56 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1205324288,
"io_kbytes" : 1177074,
"bw_bytes" : 172164589,
"bw" : 168129,
"iops" : 336258.534495,
"runtime" : 7001,
"total_ios" : 2354146,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 17002,
"mean" : 179.156082,
"stddev" : 149.002890,
"N" : 2354146
},
"clat_ns" : {
"min" : 5971,
"max" : 115316,
"mean" : 11061.462771,
"stddev" : 3262.876327,
"N" : 2354147,
"percentile" : {
"1.000000" : 6496,
"5.000000" : 7328,
"10.000000" : 7968,
"20.000000" : 8384,
"30.000000" : 8768,
"40.000000" : 9152,
"50.000000" : 10432,
"60.000000" : 12096,
"70.000000" : 12736,
"80.000000" : 13248,
"90.000000" : 13760,
"95.000000" : 17280,
"99.000000" : 21376,
"99.500000" : 24192,
"99.900000" : 30336,
"99.950000" : 35072,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 6171,
"max" : 115496,
"mean" : 11240.618823,
"stddev" : 3234.373994,
"N" : 2354147
},
"bw_min" : 166257,
"bw_max" : 170589,
"bw_agg" : 100.000000,
"bw_mean" : 168231.142857,
"bw_dev" : 1493.829661,
"bw_samples" : 14,
"iops_min" : 332515,
"iops_max" : 341178,
"iops_mean" : 336462.428571,
"iops_stddev" : 2987.544238,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 41.185714,
"sys_cpu" : 58.785714,
"ctx" : 11,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 6.693255,
"2" : 43.215119,
"4" : 50.091626,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 47.577253,
"20" : 51.221589,
"50" : 1.163182,
"100" : 0.037848,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=8974: Wed Oct 26 15:17:56 2022
read: IOPS=336k, BW=164MiB/s (172MB/s)(1149MiB/7001msec)
slat (nsec): min=110, max=17002, avg=179.16, stdev=149.00
clat (usec): min=5, max=115, avg=11.06, stdev= 3.26
lat (usec): min=6, max=115, avg=11.24, stdev= 3.23
clat percentiles (nsec):
| 1.00th=[ 6496], 5.00th=[ 7328], 10.00th=[ 7968], 20.00th=[ 8384],
| 30.00th=[ 8768], 40.00th=[ 9152], 50.00th=[10432], 60.00th=[12096],
| 70.00th=[12736], 80.00th=[13248], 90.00th=[13760], 95.00th=[17280],
| 99.00th=[21376], 99.50th=[24192], 99.90th=[30336], 99.95th=[35072],
| 99.99th=[66048]
bw ( KiB/s): min=166257, max=170589, per=100.00%, avg=168231.14, stdev=1493.83, samples=14
iops : min=332515, max=341178, avg=336462.43, stdev=2987.54, samples=14
lat (usec) : 10=47.58%, 20=51.22%, 50=1.16%, 100=0.04%, 250=0.01%
cpu : usr=41.19%, sys=58.79%, ctx=11, majf=0, minf=0
IO depths : 1=6.7%, 2=43.2%, 4=50.1%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2354146,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=164MiB/s (172MB/s), 164MiB/s-164MiB/s (172MB/s-172MB/s), io=1149MiB (1205MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789248,
"timestamp_ms" : 1666789248165,
"time" : "Wed Oct 26 15:00:48 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 407997952,
"io_kbytes" : 398435,
"bw_bytes" : 58277096,
"bw" : 56911,
"iops" : 113822.453935,
"runtime" : 7001,
"total_ios" : 796871,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 711,
"max" : 20899,
"mean" : 921.766521,
"stddev" : 69.333618,
"N" : 796871
},
"clat_ns" : {
"min" : 5120,
"max" : 80270,
"mean" : 7528.237944,
"stddev" : 1123.122707,
"N" : 796871,
"percentile" : {
"1.000000" : 6880,
"5.000000" : 7008,
"10.000000" : 7072,
"20.000000" : 7200,
"30.000000" : 7264,
"40.000000" : 7328,
"50.000000" : 7392,
"60.000000" : 7520,
"70.000000" : 7584,
"80.000000" : 7648,
"90.000000" : 7776,
"95.000000" : 7904,
"99.000000" : 11712,
"99.500000" : 14400,
"99.900000" : 16512,
"99.950000" : 17280,
"99.990000" : 60672
}
},
"lat_ns" : {
"min" : 7153,
"max" : 81202,
"mean" : 8450.004465,
"stddev" : 1126.519340,
"N" : 796871
},
"bw_min" : 56840,
"bw_max" : 57028,
"bw_agg" : 100.000000,
"bw_mean" : 56925.714286,
"bw_dev" : 58.529977,
"bw_samples" : 14,
"iops_min" : 113680,
"iops_max" : 114056,
"iops_mean" : 113851.500000,
"iops_stddev" : 117.153909,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 11,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 98.200336,
"20" : 1.771805,
"50" : 0.010000,
"100" : 0.018322,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=5285: Wed Oct 26 15:00:48 2022
read: IOPS=114k, BW=55.6MiB/s (58.3MB/s)(389MiB/7001msec)
slat (nsec): min=711, max=20899, avg=921.77, stdev=69.33
clat (nsec): min=5120, max=80270, avg=7528.24, stdev=1123.12
lat (nsec): min=7153, max=81202, avg=8450.00, stdev=1126.52
clat percentiles (nsec):
| 1.00th=[ 6880], 5.00th=[ 7008], 10.00th=[ 7072], 20.00th=[ 7200],
| 30.00th=[ 7264], 40.00th=[ 7328], 50.00th=[ 7392], 60.00th=[ 7520],
| 70.00th=[ 7584], 80.00th=[ 7648], 90.00th=[ 7776], 95.00th=[ 7904],
| 99.00th=[11712], 99.50th=[14400], 99.90th=[16512], 99.95th=[17280],
| 99.99th=[60672]
bw ( KiB/s): min=56840, max=57028, per=100.00%, avg=56925.71, stdev=58.53, samples=14
iops : min=113680, max=114056, avg=113851.50, stdev=117.15, samples=14
lat (usec) : 10=98.20%, 20=1.77%, 50=0.01%, 100=0.02%
cpu : usr=99.99%, sys=0.00%, ctx=11, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=796871,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=55.6MiB/s (58.3MB/s), 55.6MiB/s-55.6MiB/s (58.3MB/s-58.3MB/s), io=389MiB (408MB), run=7001-7001msec
$ cat /tmp/fio-output_512_4_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790291,
"timestamp_ms" : 1666790291984,
"time" : "Wed Oct 26 15:18:11 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1173137408,
"io_kbytes" : 1145642,
"bw_bytes" : 167591058,
"bw" : 163663,
"iops" : 327325.857143,
"runtime" : 7000,
"total_ios" : 2291281,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 120,
"max" : 15419,
"mean" : 174.303148,
"stddev" : 129.262014,
"N" : 2291281
},
"clat_ns" : {
"min" : 6031,
"max" : 97032,
"mean" : 11358.835785,
"stddev" : 3221.532516,
"N" : 2291284,
"percentile" : {
"1.000000" : 6496,
"5.000000" : 7392,
"10.000000" : 8032,
"20.000000" : 8512,
"30.000000" : 8896,
"40.000000" : 9664,
"50.000000" : 11712,
"60.000000" : 12608,
"70.000000" : 12992,
"80.000000" : 13376,
"90.000000" : 13760,
"95.000000" : 17536,
"99.000000" : 21376,
"99.500000" : 23936,
"99.900000" : 29568,
"99.950000" : 35072,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 6202,
"max" : 97222,
"mean" : 11533.138909,
"stddev" : 3199.841005,
"N" : 2291284
},
"bw_min" : 161805,
"bw_max" : 165426,
"bw_agg" : 100.000000,
"bw_mean" : 163730.928571,
"bw_dev" : 1280.287135,
"bw_samples" : 14,
"iops_min" : 323611,
"iops_max" : 330852,
"iops_mean" : 327461.928571,
"iops_stddev" : 2560.458567,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 42.842857,
"sys_cpu" : 57.128571,
"ctx" : 16,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 9.001777,
"2" : 44.955944,
"4" : 46.042279,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 41.212099,
"20" : 57.586259,
"50" : 1.166116,
"100" : 0.035657,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=9031: Wed Oct 26 15:18:11 2022
read: IOPS=327k, BW=160MiB/s (168MB/s)(1119MiB/7000msec)
slat (nsec): min=120, max=15419, avg=174.30, stdev=129.26
clat (nsec): min=6031, max=97032, avg=11358.84, stdev=3221.53
lat (nsec): min=6202, max=97222, avg=11533.14, stdev=3199.84
clat percentiles (nsec):
| 1.00th=[ 6496], 5.00th=[ 7392], 10.00th=[ 8032], 20.00th=[ 8512],
| 30.00th=[ 8896], 40.00th=[ 9664], 50.00th=[11712], 60.00th=[12608],
| 70.00th=[12992], 80.00th=[13376], 90.00th=[13760], 95.00th=[17536],
| 99.00th=[21376], 99.50th=[23936], 99.90th=[29568], 99.95th=[35072],
| 99.99th=[66048]
bw ( KiB/s): min=161805, max=165426, per=100.00%, avg=163730.93, stdev=1280.29, samples=14
iops : min=323611, max=330852, avg=327461.93, stdev=2560.46, samples=14
lat (usec) : 10=41.21%, 20=57.59%, 50=1.17%, 100=0.04%
cpu : usr=42.84%, sys=57.13%, ctx=16, majf=0, minf=0
IO depths : 1=9.0%, 2=45.0%, 4=46.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2291281,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=160MiB/s (168MB/s), 160MiB/s-160MiB/s (168MB/s-168MB/s), io=1119MiB (1173MB), run=7000-7000msec
$ rm /tmp/fio-output_512_4_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790307,
"timestamp_ms" : 1666790307183,
"time" : "Wed Oct 26 15:18:27 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1226106368,
"io_kbytes" : 1197369,
"bw_bytes" : 175133033,
"bw" : 171028,
"iops" : 342056.277675,
"runtime" : 7001,
"total_ios" : 2394736,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 17543,
"mean" : 182.469767,
"stddev" : 147.807901,
"N" : 2394736
},
"clat_ns" : {
"min" : 5981,
"max" : 134212,
"mean" : 10884.014790,
"stddev" : 3259.361616,
"N" : 2394738,
"percentile" : {
"1.000000" : 6496,
"5.000000" : 7392,
"10.000000" : 7968,
"20.000000" : 8384,
"30.000000" : 8768,
"40.000000" : 9024,
"50.000000" : 9664,
"60.000000" : 11712,
"70.000000" : 12608,
"80.000000" : 13120,
"90.000000" : 13760,
"95.000000" : 17536,
"99.000000" : 21376,
"99.500000" : 24192,
"99.900000" : 30080,
"99.950000" : 35072,
"99.990000" : 67072
}
},
"lat_ns" : {
"min" : 6171,
"max" : 134342,
"mean" : 11066.484526,
"stddev" : 3234.398550,
"N" : 2394738
},
"bw_min" : 167302,
"bw_max" : 174220,
"bw_agg" : 100.000000,
"bw_mean" : 171056.571429,
"bw_dev" : 2202.020531,
"bw_samples" : 14,
"iops_min" : 334604,
"iops_max" : 348440,
"iops_mean" : 342113.142857,
"iops_stddev" : 4404.041061,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 38.157143,
"sys_cpu" : 61.814286,
"ctx" : 13,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 5.818804,
"2" : 42.767136,
"4" : 51.414060,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 51.271330,
"20" : 47.557309,
"50" : 1.134739,
"100" : 0.035996,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=9088: Wed Oct 26 15:18:27 2022
read: IOPS=342k, BW=167MiB/s (175MB/s)(1169MiB/7001msec)
slat (nsec): min=110, max=17543, avg=182.47, stdev=147.81
clat (usec): min=5, max=134, avg=10.88, stdev= 3.26
lat (usec): min=6, max=134, avg=11.07, stdev= 3.23
clat percentiles (nsec):
| 1.00th=[ 6496], 5.00th=[ 7392], 10.00th=[ 7968], 20.00th=[ 8384],
| 30.00th=[ 8768], 40.00th=[ 9024], 50.00th=[ 9664], 60.00th=[11712],
| 70.00th=[12608], 80.00th=[13120], 90.00th=[13760], 95.00th=[17536],
| 99.00th=[21376], 99.50th=[24192], 99.90th=[30080], 99.95th=[35072],
| 99.99th=[67072]
bw ( KiB/s): min=167302, max=174220, per=100.00%, avg=171056.57, stdev=2202.02, samples=14
iops : min=334604, max=348440, avg=342113.14, stdev=4404.04, samples=14
lat (usec) : 10=51.27%, 20=47.56%, 50=1.13%, 100=0.04%, 250=0.01%
cpu : usr=38.16%, sys=61.81%, ctx=13, majf=0, minf=0
IO depths : 1=5.8%, 2=42.8%, 4=51.4%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2394736,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=1169MiB (1226MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790322,
"timestamp_ms" : 1666790322440,
"time" : "Wed Oct 26 15:18:42 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1559368192,
"io_kbytes" : 1522820,
"bw_bytes" : 222735065,
"bw" : 217514,
"iops" : 435028.853021,
"runtime" : 7001,
"total_ios" : 3045637,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1603,
"max" : 68709,
"mean" : 1767.973366,
"stddev" : 121.396579,
"N" : 3045638
},
"clat_ns" : {
"min" : 220,
"max" : 119765,
"mean" : 7169.615502,
"stddev" : 2186.549003,
"N" : 3045640,
"percentile" : {
"1.000000" : 6624,
"5.000000" : 6688,
"10.000000" : 6688,
"20.000000" : 6688,
"30.000000" : 6752,
"40.000000" : 6752,
"50.000000" : 6752,
"60.000000" : 6816,
"70.000000" : 6816,
"80.000000" : 6880,
"90.000000" : 7072,
"95.000000" : 7328,
"99.000000" : 18816,
"99.500000" : 21888,
"99.900000" : 25216,
"99.950000" : 32128,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 5411,
"max" : 121538,
"mean" : 8937.588861,
"stddev" : 2192.642066,
"N" : 3045640
},
"bw_min" : 216461,
"bw_max" : 219449,
"bw_agg" : 100.000000,
"bw_mean" : 217555.357143,
"bw_dev" : 811.925310,
"bw_samples" : 14,
"iops_min" : 432922,
"iops_max" : 438898,
"iops_mean" : 435110.714286,
"iops_stddev" : 1623.850620,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 29.928571,
"sys_cpu" : 70.042857,
"ctx" : 17,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 95.836306,
"20" : 3.568055,
"50" : 0.557814,
"100" : 0.036511,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=9145: Wed Oct 26 15:18:42 2022
read: IOPS=435k, BW=212MiB/s (223MB/s)(1487MiB/7001msec)
slat (nsec): min=1603, max=68709, avg=1767.97, stdev=121.40
clat (nsec): min=220, max=119765, avg=7169.62, stdev=2186.55
lat (usec): min=5, max=121, avg= 8.94, stdev= 2.19
clat percentiles (nsec):
| 1.00th=[ 6624], 5.00th=[ 6688], 10.00th=[ 6688], 20.00th=[ 6688],
| 30.00th=[ 6752], 40.00th=[ 6752], 50.00th=[ 6752], 60.00th=[ 6816],
| 70.00th=[ 6816], 80.00th=[ 6880], 90.00th=[ 7072], 95.00th=[ 7328],
| 99.00th=[18816], 99.50th=[21888], 99.90th=[25216], 99.95th=[32128],
| 99.99th=[66048]
bw ( KiB/s): min=216461, max=219449, per=100.00%, avg=217555.36, stdev=811.93, samples=14
iops : min=432922, max=438898, avg=435110.71, stdev=1623.85, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=95.84%, 20=3.57%, 50=0.56%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=29.93%, sys=70.04%, ctx=17, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3045637,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=212MiB/s (223MB/s), 212MiB/s-212MiB/s (223MB/s-223MB/s), io=1487MiB (1559MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790337,
"timestamp_ms" : 1666790337589,
"time" : "Wed Oct 26 15:18:57 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1562582528,
"io_kbytes" : 1525959,
"bw_bytes" : 223194190,
"bw" : 217963,
"iops" : 435925.582060,
"runtime" : 7001,
"total_ios" : 3051915,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1593,
"max" : 23705,
"mean" : 1775.126412,
"stddev" : 116.995816,
"N" : 3051916
},
"clat_ns" : {
"min" : 231,
"max" : 117450,
"mean" : 7161.374734,
"stddev" : 2248.409210,
"N" : 3051918,
"percentile" : {
"1.000000" : 6624,
"5.000000" : 6624,
"10.000000" : 6688,
"20.000000" : 6688,
"30.000000" : 6752,
"40.000000" : 6752,
"50.000000" : 6752,
"60.000000" : 6752,
"70.000000" : 6816,
"80.000000" : 6880,
"90.000000" : 7072,
"95.000000" : 7264,
"99.000000" : 19328,
"99.500000" : 24192,
"99.900000" : 25728,
"99.950000" : 32128,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 5370,
"max" : 119194,
"mean" : 8936.501126,
"stddev" : 2254.690146,
"N" : 3051918
},
"bw_min" : 216970,
"bw_max" : 219040,
"bw_agg" : 100.000000,
"bw_mean" : 218076.500000,
"bw_dev" : 597.526279,
"bw_samples" : 14,
"iops_min" : 433940,
"iops_max" : 438080,
"iops_mean" : 436153.071429,
"iops_stddev" : 1194.975537,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 30.000000,
"sys_cpu" : 69.985714,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.118994,
"20" : 3.222043,
"50" : 0.619414,
"100" : 0.035486,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=9203: Wed Oct 26 15:18:57 2022
read: IOPS=436k, BW=213MiB/s (223MB/s)(1490MiB/7001msec)
slat (nsec): min=1593, max=23705, avg=1775.13, stdev=117.00
clat (nsec): min=231, max=117450, avg=7161.37, stdev=2248.41
lat (usec): min=5, max=119, avg= 8.94, stdev= 2.25
clat percentiles (nsec):
| 1.00th=[ 6624], 5.00th=[ 6624], 10.00th=[ 6688], 20.00th=[ 6688],
| 30.00th=[ 6752], 40.00th=[ 6752], 50.00th=[ 6752], 60.00th=[ 6752],
| 70.00th=[ 6816], 80.00th=[ 6880], 90.00th=[ 7072], 95.00th=[ 7264],
| 99.00th=[19328], 99.50th=[24192], 99.90th=[25728], 99.95th=[32128],
| 99.99th=[66048]
bw ( KiB/s): min=216970, max=219040, per=100.00%, avg=218076.50, stdev=597.53, samples=14
iops : min=433940, max=438080, avg=436153.07, stdev=1194.98, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=96.12%, 20=3.22%, 50=0.62%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=30.00%, sys=69.99%, ctx=10, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3051915,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=213MiB/s (223MB/s), 213MiB/s-213MiB/s (223MB/s-223MB/s), io=1490MiB (1563MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory
$ rm /tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790352,
"timestamp_ms" : 1666790352617,
"time" : "Wed Oct 26 15:19:12 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1559137792,
"io_kbytes" : 1522595,
"bw_bytes" : 222702155,
"bw" : 217482,
"iops" : 434964.576489,
"runtime" : 7001,
"total_ios" : 3045187,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1593,
"max" : 20638,
"mean" : 1801.881652,
"stddev" : 112.241295,
"N" : 3045188
},
"clat_ns" : {
"min" : 211,
"max" : 117270,
"mean" : 7169.162602,
"stddev" : 2248.200291,
"N" : 3045190,
"percentile" : {
"1.000000" : 6624,
"5.000000" : 6624,
"10.000000" : 6688,
"20.000000" : 6688,
"30.000000" : 6752,
"40.000000" : 6752,
"50.000000" : 6752,
"60.000000" : 6816,
"70.000000" : 6816,
"80.000000" : 6880,
"90.000000" : 7072,
"95.000000" : 7264,
"99.000000" : 19328,
"99.500000" : 24192,
"99.900000" : 25728,
"99.950000" : 32384,
"99.990000" : 67072
}
},
"lat_ns" : {
"min" : 5360,
"max" : 118993,
"mean" : 8971.044281,
"stddev" : 2253.640954,
"N" : 3045190
},
"bw_min" : 216412,
"bw_max" : 218927,
"bw_agg" : 100.000000,
"bw_mean" : 217521.785714,
"bw_dev" : 661.670750,
"bw_samples" : 14,
"iops_min" : 432824,
"iops_max" : 437854,
"iops_mean" : 435043.642857,
"iops_stddev" : 1323.496628,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 32.385714,
"sys_cpu" : 67.600000,
"ctx" : 14,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.063559,
"20" : 3.301308,
"50" : 0.594709,
"100" : 0.036911,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=9260: Wed Oct 26 15:19:12 2022
read: IOPS=435k, BW=212MiB/s (223MB/s)(1487MiB/7001msec)
slat (nsec): min=1593, max=20638, avg=1801.88, stdev=112.24
clat (nsec): min=211, max=117270, avg=7169.16, stdev=2248.20
lat (usec): min=5, max=118, avg= 8.97, stdev= 2.25
clat percentiles (nsec):
| 1.00th=[ 6624], 5.00th=[ 6624], 10.00th=[ 6688], 20.00th=[ 6688],
| 30.00th=[ 6752], 40.00th=[ 6752], 50.00th=[ 6752], 60.00th=[ 6816],
| 70.00th=[ 6816], 80.00th=[ 6880], 90.00th=[ 7072], 95.00th=[ 7264],
| 99.00th=[19328], 99.50th=[24192], 99.90th=[25728], 99.95th=[32384],
| 99.99th=[67072]
bw ( KiB/s): min=216412, max=218927, per=100.00%, avg=217521.79, stdev=661.67, samples=14
iops : min=432824, max=437854, avg=435043.64, stdev=1323.50, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=96.06%, 20=3.30%, 50=0.59%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=32.39%, sys=67.60%, ctx=14, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3045187,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=212MiB/s (223MB/s), 212MiB/s-212MiB/s (223MB/s-223MB/s), io=1487MiB (1559MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790366,
"timestamp_ms" : 1666790366377,
"time" : "Wed Oct 26 15:19:26 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1420984832,
"io_kbytes" : 1387680,
"bw_bytes" : 202968837,
"bw" : 198211,
"iops" : 396423.082417,
"runtime" : 7001,
"total_ios" : 2775358,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1382,
"max" : 67266,
"mean" : 1980.791384,
"stddev" : 330.353282,
"N" : 2775358
},
"clat_ns" : {
"min" : 2054,
"max" : 124162,
"mean" : 7810.983699,
"stddev" : 2690.900419,
"N" : 2775361,
"percentile" : {
"1.000000" : 4448,
"5.000000" : 5152,
"10.000000" : 6816,
"20.000000" : 6944,
"30.000000" : 7392,
"40.000000" : 7456,
"50.000000" : 7456,
"60.000000" : 7520,
"70.000000" : 7648,
"80.000000" : 8032,
"90.000000" : 8384,
"95.000000" : 9408,
"99.000000" : 20864,
"99.500000" : 25216,
"99.900000" : 27264,
"99.950000" : 33536,
"99.990000" : 69120
}
},
"lat_ns" : {
"min" : 5781,
"max" : 126286,
"mean" : 9791.775021,
"stddev" : 2666.740323,
"N" : 2775361
},
"bw_min" : 197094,
"bw_max" : 199345,
"bw_agg" : 100.000000,
"bw_mean" : 198313.214286,
"bw_dev" : 631.854801,
"bw_samples" : 14,
"iops_min" : 394188,
"iops_max" : 398690,
"iops_mean" : 396626.571429,
"iops_stddev" : 1263.775583,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.671429,
"sys_cpu" : 78.628571,
"ctx" : 27529,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.050924,
"20" : 2.666179,
"50" : 1.239480,
"100" : 0.041184,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=9307: Wed Oct 26 15:19:26 2022
read: IOPS=396k, BW=194MiB/s (203MB/s)(1355MiB/7001msec)
slat (nsec): min=1382, max=67266, avg=1980.79, stdev=330.35
clat (usec): min=2, max=124, avg= 7.81, stdev= 2.69
lat (usec): min=5, max=126, avg= 9.79, stdev= 2.67
clat percentiles (nsec):
| 1.00th=[ 4448], 5.00th=[ 5152], 10.00th=[ 6816], 20.00th=[ 6944],
| 30.00th=[ 7392], 40.00th=[ 7456], 50.00th=[ 7456], 60.00th=[ 7520],
| 70.00th=[ 7648], 80.00th=[ 8032], 90.00th=[ 8384], 95.00th=[ 9408],
| 99.00th=[20864], 99.50th=[25216], 99.90th=[27264], 99.95th=[33536],
| 99.99th=[69120]
bw ( KiB/s): min=197094, max=199345, per=100.00%, avg=198313.21, stdev=631.85, samples=14
iops : min=394188, max=398690, avg=396626.57, stdev=1263.78, samples=14
lat (usec) : 4=0.01%, 10=96.05%, 20=2.67%, 50=1.24%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=17.67%, sys=78.63%, ctx=27529, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2775358,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=194MiB/s (203MB/s), 194MiB/s-194MiB/s (203MB/s-203MB/s), io=1355MiB (1421MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790380,
"timestamp_ms" : 1666790380127,
"time" : "Wed Oct 26 15:19:40 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1402958336,
"io_kbytes" : 1370076,
"bw_bytes" : 200393991,
"bw" : 195697,
"iops" : 391393.943722,
"runtime" : 7001,
"total_ios" : 2740149,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1383,
"max" : 22943,
"mean" : 2020.769172,
"stddev" : 287.667604,
"N" : 2740150
},
"clat_ns" : {
"min" : 2103,
"max" : 122420,
"mean" : 7908.474523,
"stddev" : 2587.271863,
"N" : 2740152,
"percentile" : {
"1.000000" : 4512,
"5.000000" : 6816,
"10.000000" : 6880,
"20.000000" : 7456,
"30.000000" : 7456,
"40.000000" : 7520,
"50.000000" : 7520,
"60.000000" : 7584,
"70.000000" : 7648,
"80.000000" : 7840,
"90.000000" : 8256,
"95.000000" : 9024,
"99.000000" : 20608,
"99.500000" : 24960,
"99.900000" : 27008,
"99.950000" : 33024,
"99.990000" : 69120
}
},
"lat_ns" : {
"min" : 5590,
"max" : 123922,
"mean" : 9929.243570,
"stddev" : 2555.049304,
"N" : 2740152
},
"bw_min" : 194510,
"bw_max" : 197872,
"bw_agg" : 100.000000,
"bw_mean" : 195731.571429,
"bw_dev" : 1047.071059,
"bw_samples" : 14,
"iops_min" : 389020,
"iops_max" : 395744,
"iops_mean" : 391463.214286,
"iops_stddev" : 2094.197044,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 20.785714,
"sys_cpu" : 75.628571,
"ctx" : 27189,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.072622,
"20" : 2.739012,
"50" : 1.149791,
"100" : 0.037115,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=9355: Wed Oct 26 15:19:40 2022
read: IOPS=391k, BW=191MiB/s (200MB/s)(1338MiB/7001msec)
slat (nsec): min=1383, max=22943, avg=2020.77, stdev=287.67
clat (usec): min=2, max=122, avg= 7.91, stdev= 2.59
lat (usec): min=5, max=123, avg= 9.93, stdev= 2.56
clat percentiles (nsec):
| 1.00th=[ 4512], 5.00th=[ 6816], 10.00th=[ 6880], 20.00th=[ 7456],
| 30.00th=[ 7456], 40.00th=[ 7520], 50.00th=[ 7520], 60.00th=[ 7584],
| 70.00th=[ 7648], 80.00th=[ 7840], 90.00th=[ 8256], 95.00th=[ 9024],
| 99.00th=[20608], 99.50th=[24960], 99.90th=[27008], 99.95th=[33024],
| 99.99th=[69120]
bw ( KiB/s): min=194510, max=197872, per=100.00%, avg=195731.57, stdev=1047.07, samples=14
iops : min=389020, max=395744, avg=391463.21, stdev=2094.20, samples=14
lat (usec) : 4=0.01%, 10=96.07%, 20=2.74%, 50=1.15%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=20.79%, sys=75.63%, ctx=27189, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2740149,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=191MiB/s (200MB/s), 191MiB/s-191MiB/s (200MB/s-200MB/s), io=1338MiB (1403MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790393,
"timestamp_ms" : 1666790393982,
"time" : "Wed Oct 26 15:19:53 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1408428544,
"io_kbytes" : 1375418,
"bw_bytes" : 201175338,
"bw" : 196460,
"iops" : 392920.011427,
"runtime" : 7001,
"total_ios" : 2750833,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1402,
"max" : 43572,
"mean" : 2012.219970,
"stddev" : 307.976683,
"N" : 2750834
},
"clat_ns" : {
"min" : 131,
"max" : 104275,
"mean" : 7874.613087,
"stddev" : 2621.702740,
"N" : 2750836,
"percentile" : {
"1.000000" : 4448,
"5.000000" : 6304,
"10.000000" : 6880,
"20.000000" : 7200,
"30.000000" : 7456,
"40.000000" : 7520,
"50.000000" : 7520,
"60.000000" : 7584,
"70.000000" : 7648,
"80.000000" : 7904,
"90.000000" : 8256,
"95.000000" : 9280,
"99.000000" : 20864,
"99.500000" : 25216,
"99.900000" : 27008,
"99.950000" : 33024,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 5771,
"max" : 106169,
"mean" : 9886.833346,
"stddev" : 2594.294429,
"N" : 2750836
},
"bw_min" : 195388,
"bw_max" : 198951,
"bw_agg" : 100.000000,
"bw_mean" : 196495.071429,
"bw_dev" : 1111.164217,
"bw_samples" : 14,
"iops_min" : 390776,
"iops_max" : 397903,
"iops_mean" : 392990.214286,
"iops_stddev" : 2222.498462,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 21.085714,
"sys_cpu" : 75.314286,
"ctx" : 27181,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.066900,
"20" : 2.703181,
"50" : 1.190040,
"100" : 0.038025,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=9402: Wed Oct 26 15:19:53 2022
read: IOPS=393k, BW=192MiB/s (201MB/s)(1343MiB/7001msec)
slat (nsec): min=1402, max=43572, avg=2012.22, stdev=307.98
clat (nsec): min=131, max=104275, avg=7874.61, stdev=2621.70
lat (usec): min=5, max=106, avg= 9.89, stdev= 2.59
clat percentiles (nsec):
| 1.00th=[ 4448], 5.00th=[ 6304], 10.00th=[ 6880], 20.00th=[ 7200],
| 30.00th=[ 7456], 40.00th=[ 7520], 50.00th=[ 7520], 60.00th=[ 7584],
| 70.00th=[ 7648], 80.00th=[ 7904], 90.00th=[ 8256], 95.00th=[ 9280],
| 99.00th=[20864], 99.50th=[25216], 99.90th=[27008], 99.95th=[33024],
| 99.99th=[68096]
bw ( KiB/s): min=195388, max=198951, per=100.00%, avg=196495.07, stdev=1111.16, samples=14
iops : min=390776, max=397903, avg=392990.21, stdev=2222.50, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=96.07%, 20=2.70%, 50=1.19%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=21.09%, sys=75.31%, ctx=27181, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2750833,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=192MiB/s (201MB/s), 192MiB/s-192MiB/s (201MB/s-201MB/s), io=1343MiB (1408MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790407,
"timestamp_ms" : 1666790407746,
"time" : "Wed Oct 26 15:20:07 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1540268544,
"io_kbytes" : 1504168,
"bw_bytes" : 220006933,
"bw" : 214850,
"iops" : 429700.471361,
"runtime" : 7001,
"total_ios" : 3008333,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1442,
"max" : 22633,
"mean" : 1606.975883,
"stddev" : 117.237640,
"N" : 3008334
},
"clat_ns" : {
"min" : 2945,
"max" : 110618,
"mean" : 7468.559953,
"stddev" : 2665.426037,
"N" : 3008336,
"percentile" : {
"1.000000" : 6816,
"5.000000" : 6880,
"10.000000" : 6880,
"20.000000" : 6944,
"30.000000" : 6944,
"40.000000" : 6944,
"50.000000" : 7008,
"60.000000" : 7008,
"70.000000" : 7072,
"80.000000" : 7200,
"90.000000" : 7264,
"95.000000" : 7392,
"99.000000" : 22400,
"99.500000" : 27008,
"99.900000" : 32640,
"99.950000" : 35072,
"99.990000" : 69120
}
},
"lat_ns" : {
"min" : 6051,
"max" : 112181,
"mean" : 9075.535866,
"stddev" : 2669.379295,
"N" : 3008336
},
"bw_min" : 213880,
"bw_max" : 215909,
"bw_agg" : 100.000000,
"bw_mean" : 214911.214286,
"bw_dev" : 609.982620,
"bw_samples" : 14,
"iops_min" : 427760,
"iops_max" : 431818,
"iops_mean" : 429822.428571,
"iops_stddev" : 1219.965240,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 28.971429,
"sys_cpu" : 67.200000,
"ctx" : 26583,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.583922,
"20" : 2.229574,
"50" : 1.152033,
"100" : 0.034438,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 4333182,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 23213,
"write_ticks" : 0,
"in_queue" : 23213,
"util" : 99.069123
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=9449: Wed Oct 26 15:20:07 2022
read: IOPS=430k, BW=210MiB/s (220MB/s)(1469MiB/7001msec)
slat (nsec): min=1442, max=22633, avg=1606.98, stdev=117.24
clat (usec): min=2, max=110, avg= 7.47, stdev= 2.67
lat (usec): min=6, max=112, avg= 9.08, stdev= 2.67
clat percentiles (nsec):
| 1.00th=[ 6816], 5.00th=[ 6880], 10.00th=[ 6880], 20.00th=[ 6944],
| 30.00th=[ 6944], 40.00th=[ 6944], 50.00th=[ 7008], 60.00th=[ 7008],
| 70.00th=[ 7072], 80.00th=[ 7200], 90.00th=[ 7264], 95.00th=[ 7392],
| 99.00th=[22400], 99.50th=[27008], 99.90th=[32640], 99.95th=[35072],
| 99.99th=[69120]
bw ( KiB/s): min=213880, max=215909, per=100.00%, avg=214911.21, stdev=609.98, samples=14
iops : min=427760, max=431818, avg=429822.43, stdev=1219.97, samples=14
lat (usec) : 4=0.01%, 10=96.58%, 20=2.23%, 50=1.15%, 100=0.03%
lat (usec) : 250=0.01%
cpu : usr=28.97%, sys=67.20%, ctx=26583, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3008333,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=210MiB/s (220MB/s), 210MiB/s-210MiB/s (220MB/s-220MB/s), io=1469MiB (1540MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=4333182/0, merge=0/0, ticks=23213/0, in_queue=23213, util=99.07%
$ rm /tmp/fio-output_512_4_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790421,
"timestamp_ms" : 1666790421560,
"time" : "Wed Oct 26 15:20:21 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1547104768,
"io_kbytes" : 1510844,
"bw_bytes" : 220983397,
"bw" : 215804,
"iops" : 431607.627482,
"runtime" : 7001,
"total_ios" : 3021685,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1433,
"max" : 30357,
"mean" : 1591.780563,
"stddev" : 110.756728,
"N" : 3021686
},
"clat_ns" : {
"min" : 4548,
"max" : 126658,
"mean" : 7446.293310,
"stddev" : 2622.424351,
"N" : 3021688,
"percentile" : {
"1.000000" : 6880,
"5.000000" : 6880,
"10.000000" : 6944,
"20.000000" : 6944,
"30.000000" : 6944,
"40.000000" : 6944,
"50.000000" : 7008,
"60.000000" : 7008,
"70.000000" : 7008,
"80.000000" : 7072,
"90.000000" : 7264,
"95.000000" : 7392,
"99.000000" : 21120,
"99.500000" : 24960,
"99.900000" : 31872,
"99.950000" : 35072,
"99.990000" : 70144
}
},
"lat_ns" : {
"min" : 6021,
"max" : 128201,
"mean" : 9038.073877,
"stddev" : 2623.736564,
"N" : 3021688
},
"bw_min" : 215056,
"bw_max" : 217372,
"bw_agg" : 100.000000,
"bw_mean" : 215893.642857,
"bw_dev" : 579.696420,
"bw_samples" : 14,
"iops_min" : 430112,
"iops_max" : 434745,
"iops_mean" : 431787.571429,
"iops_stddev" : 1159.639978,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 32.028571,
"sys_cpu" : 64.071429,
"ctx" : 29587,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.336084,
"20" : 2.499665,
"50" : 1.126226,
"100" : 0.037760,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 4350489,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 21803,
"write_ticks" : 0,
"in_queue" : 21803,
"util" : 99.078934
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=9497: Wed Oct 26 15:20:21 2022
read: IOPS=432k, BW=211MiB/s (221MB/s)(1475MiB/7001msec)
slat (nsec): min=1433, max=30357, avg=1591.78, stdev=110.76
clat (usec): min=4, max=126, avg= 7.45, stdev= 2.62
lat (usec): min=6, max=128, avg= 9.04, stdev= 2.62
clat percentiles (nsec):
| 1.00th=[ 6880], 5.00th=[ 6880], 10.00th=[ 6944], 20.00th=[ 6944],
| 30.00th=[ 6944], 40.00th=[ 6944], 50.00th=[ 7008], 60.00th=[ 7008],
| 70.00th=[ 7008], 80.00th=[ 7072], 90.00th=[ 7264], 95.00th=[ 7392],
| 99.00th=[21120], 99.50th=[24960], 99.90th=[31872], 99.95th=[35072],
| 99.99th=[70144]
bw ( KiB/s): min=215056, max=217372, per=100.00%, avg=215893.64, stdev=579.70, samples=14
iops : min=430112, max=434745, avg=431787.57, stdev=1159.64, samples=14
lat (usec) : 10=96.34%, 20=2.50%, 50=1.13%, 100=0.04%, 250=0.01%
cpu : usr=32.03%, sys=64.07%, ctx=29587, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3021685,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=211MiB/s (221MB/s), 211MiB/s-211MiB/s (221MB/s-221MB/s), io=1475MiB (1547MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=4350489/0, merge=0/0, ticks=21803/0, in_queue=21803, util=99.08%
$ rm /tmp/fio-output_512_4_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789263,
"timestamp_ms" : 1666789263383,
"time" : "Wed Oct 26 15:01:03 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 507951104,
"io_kbytes" : 496046,
"bw_bytes" : 72564443,
"bw" : 70863,
"iops" : 141727.428571,
"runtime" : 7000,
"total_ios" : 992092,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 250,
"max" : 25638,
"mean" : 278.643752,
"stddev" : 68.503384,
"N" : 992092
},
"clat_ns" : {
"min" : 320,
"max" : 78928,
"mean" : 6484.535887,
"stddev" : 1090.312041,
"N" : 992092,
"percentile" : {
"1.000000" : 5984,
"5.000000" : 6176,
"10.000000" : 6240,
"20.000000" : 6240,
"30.000000" : 6304,
"40.000000" : 6304,
"50.000000" : 6368,
"60.000000" : 6368,
"70.000000" : 6432,
"80.000000" : 6432,
"90.000000" : 6496,
"95.000000" : 6624,
"99.000000" : 11456,
"99.500000" : 13504,
"99.900000" : 17024,
"99.950000" : 19328,
"99.990000" : 58112
}
},
"lat_ns" : {
"min" : 5991,
"max" : 79199,
"mean" : 6763.179639,
"stddev" : 1092.110784,
"N" : 992092
},
"bw_min" : 70687,
"bw_max" : 71173,
"bw_agg" : 100.000000,
"bw_mean" : 70885.071429,
"bw_dev" : 141.145023,
"bw_samples" : 14,
"iops_min" : 141374,
"iops_max" : 142346,
"iops_mean" : 141770.285714,
"iops_stddev" : 282.276032,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 98.826016,
"20" : 1.129028,
"50" : 0.031953,
"100" : 0.012398,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=5343: Wed Oct 26 15:01:03 2022
read: IOPS=142k, BW=69.2MiB/s (72.6MB/s)(484MiB/7000msec)
slat (nsec): min=250, max=25638, avg=278.64, stdev=68.50
clat (nsec): min=320, max=78928, avg=6484.54, stdev=1090.31
lat (nsec): min=5991, max=79199, avg=6763.18, stdev=1092.11
clat percentiles (nsec):
| 1.00th=[ 5984], 5.00th=[ 6176], 10.00th=[ 6240], 20.00th=[ 6240],
| 30.00th=[ 6304], 40.00th=[ 6304], 50.00th=[ 6368], 60.00th=[ 6368],
| 70.00th=[ 6432], 80.00th=[ 6432], 90.00th=[ 6496], 95.00th=[ 6624],
| 99.00th=[11456], 99.50th=[13504], 99.90th=[17024], 99.95th=[19328],
| 99.99th=[58112]
bw ( KiB/s): min=70687, max=71173, per=100.00%, avg=70885.07, stdev=141.15, samples=14
iops : min=141374, max=142346, avg=141770.29, stdev=282.28, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 10=98.83%, 20=1.13%, 50=0.03%, 100=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=10, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=992092,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=69.2MiB/s (72.6MB/s), 69.2MiB/s-69.2MiB/s (72.6MB/s-72.6MB/s), io=484MiB (508MB), run=7000-7000msec
$ cat /tmp/fio-output_512_4_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790435,
"timestamp_ms" : 1666790435451,
"time" : "Wed Oct 26 15:20:35 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1523316736,
"io_kbytes" : 1487614,
"bw_bytes" : 217585592,
"bw" : 212485,
"iops" : 424971.289816,
"runtime" : 7001,
"total_ios" : 2975224,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1453,
"max" : 26480,
"mean" : 1626.616954,
"stddev" : 111.977939,
"N" : 2975225
},
"clat_ns" : {
"min" : 4588,
"max" : 122941,
"mean" : 7550.090039,
"stddev" : 2701.313923,
"N" : 2975227,
"percentile" : {
"1.000000" : 6944,
"5.000000" : 6944,
"10.000000" : 7008,
"20.000000" : 7008,
"30.000000" : 7008,
"40.000000" : 7072,
"50.000000" : 7072,
"60.000000" : 7072,
"70.000000" : 7136,
"80.000000" : 7328,
"90.000000" : 7392,
"95.000000" : 7456,
"99.000000" : 22400,
"99.500000" : 27008,
"99.900000" : 33024,
"99.950000" : 38656,
"99.990000" : 71168
}
},
"lat_ns" : {
"min" : 6101,
"max" : 124514,
"mean" : 9176.707014,
"stddev" : 2705.260929,
"N" : 2975227
},
"bw_min" : 212046,
"bw_max" : 213823,
"bw_agg" : 100.000000,
"bw_mean" : 212606.928571,
"bw_dev" : 548.418771,
"bw_samples" : 14,
"iops_min" : 424092,
"iops_max" : 427646,
"iops_mean" : 425213.928571,
"iops_stddev" : 1096.881206,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 34.171429,
"sys_cpu" : 62.028571,
"ctx" : 26352,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.603953,
"20" : 2.192776,
"50" : 1.162669,
"100" : 0.040568,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 4292890,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 22927,
"write_ticks" : 0,
"in_queue" : 22927,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=9543: Wed Oct 26 15:20:35 2022
read: IOPS=425k, BW=208MiB/s (218MB/s)(1453MiB/7001msec)
slat (nsec): min=1453, max=26480, avg=1626.62, stdev=111.98
clat (usec): min=4, max=122, avg= 7.55, stdev= 2.70
lat (usec): min=6, max=124, avg= 9.18, stdev= 2.71
clat percentiles (nsec):
| 1.00th=[ 6944], 5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7008],
| 30.00th=[ 7008], 40.00th=[ 7072], 50.00th=[ 7072], 60.00th=[ 7072],
| 70.00th=[ 7136], 80.00th=[ 7328], 90.00th=[ 7392], 95.00th=[ 7456],
| 99.00th=[22400], 99.50th=[27008], 99.90th=[33024], 99.95th=[38656],
| 99.99th=[71168]
bw ( KiB/s): min=212046, max=213823, per=100.00%, avg=212606.93, stdev=548.42, samples=14
iops : min=424092, max=427646, avg=425213.93, stdev=1096.88, samples=14
lat (usec) : 10=96.60%, 20=2.19%, 50=1.16%, 100=0.04%, 250=0.01%
cpu : usr=34.17%, sys=62.03%, ctx=26352, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2975224,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=208MiB/s (218MB/s), 208MiB/s-208MiB/s (218MB/s-218MB/s), io=1453MiB (1523MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=4292890/0, merge=0/0, ticks=22927/0, in_queue=22927, util=99.09%
$ rm /tmp/fio-output_512_4_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-bdev_aio_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790450,
"timestamp_ms" : 1666790450686,
"time" : "Wed Oct 26 15:20:50 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1562015232,
"io_kbytes" : 1525405,
"bw_bytes" : 223113159,
"bw" : 217883,
"iops" : 435767.318954,
"runtime" : 7001,
"total_ios" : 3050807,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1623,
"max" : 24445,
"mean" : 1823.933966,
"stddev" : 122.188443,
"N" : 3050808
},
"clat_ns" : {
"min" : 271,
"max" : 121558,
"mean" : 6479.020192,
"stddev" : 2349.697327,
"N" : 3050807,
"percentile" : {
"1.000000" : 4384,
"5.000000" : 4448,
"10.000000" : 4576,
"20.000000" : 4704,
"30.000000" : 6496,
"40.000000" : 6624,
"50.000000" : 6688,
"60.000000" : 6752,
"70.000000" : 6752,
"80.000000" : 6816,
"90.000000" : 6944,
"95.000000" : 7136,
"99.000000" : 17536,
"99.500000" : 20352,
"99.900000" : 25472,
"99.950000" : 30848,
"99.990000" : 65280
}
},
"lat_ns" : {
"min" : 5370,
"max" : 123371,
"mean" : 8302.954171,
"stddev" : 2352.126482,
"N" : 3050807
},
"bw_min" : 216213,
"bw_max" : 219913,
"bw_agg" : 100.000000,
"bw_mean" : 218014.214286,
"bw_dev" : 1097.446559,
"bw_samples" : 14,
"iops_min" : 432426,
"iops_max" : 439826,
"iops_mean" : 436028.714286,
"iops_stddev" : 2194.874043,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 38.500000,
"sys_cpu" : 61.471429,
"ctx" : 14,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 29.427558,
"4" : 70.569525,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.818940,
"20" : 2.569976,
"50" : 0.567129,
"100" : 0.034483,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=9600: Wed Oct 26 15:20:50 2022
read: IOPS=436k, BW=213MiB/s (223MB/s)(1490MiB/7001msec)
slat (nsec): min=1623, max=24445, avg=1823.93, stdev=122.19
clat (nsec): min=271, max=121558, avg=6479.02, stdev=2349.70
lat (usec): min=5, max=123, avg= 8.30, stdev= 2.35
clat percentiles (nsec):
| 1.00th=[ 4384], 5.00th=[ 4448], 10.00th=[ 4576], 20.00th=[ 4704],
| 30.00th=[ 6496], 40.00th=[ 6624], 50.00th=[ 6688], 60.00th=[ 6752],
| 70.00th=[ 6752], 80.00th=[ 6816], 90.00th=[ 6944], 95.00th=[ 7136],
| 99.00th=[17536], 99.50th=[20352], 99.90th=[25472], 99.95th=[30848],
| 99.99th=[65280]
bw ( KiB/s): min=216213, max=219913, per=100.00%, avg=218014.21, stdev=1097.45, samples=14
iops : min=432426, max=439826, avg=436028.71, stdev=2194.87, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=96.82%, 20=2.57%, 50=0.57%, 100=0.03%
lat (usec) : 250=0.01%
cpu : usr=38.50%, sys=61.47%, ctx=14, majf=0, minf=0
IO depths : 1=0.1%, 2=29.4%, 4=70.6%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3050807,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=213MiB/s (223MB/s), 213MiB/s-213MiB/s (223MB/s-223MB/s), io=1490MiB (1562MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-bdev_aio_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790466,
"timestamp_ms" : 1666790466054,
"time" : "Wed Oct 26 15:21:06 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1556518912,
"io_kbytes" : 1520038,
"bw_bytes" : 222328083,
"bw" : 217117,
"iops" : 434234.109413,
"runtime" : 7001,
"total_ios" : 3040073,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1643,
"max" : 40897,
"mean" : 1833.893626,
"stddev" : 124.135208,
"N" : 3040073
},
"clat_ns" : {
"min" : 290,
"max" : 124533,
"mean" : 6469.387454,
"stddev" : 2382.210011,
"N" : 3040076,
"percentile" : {
"1.000000" : 4448,
"5.000000" : 4448,
"10.000000" : 4576,
"20.000000" : 4704,
"30.000000" : 6432,
"40.000000" : 6624,
"50.000000" : 6688,
"60.000000" : 6752,
"70.000000" : 6816,
"80.000000" : 6816,
"90.000000" : 6944,
"95.000000" : 7200,
"99.000000" : 17536,
"99.500000" : 20352,
"99.900000" : 25728,
"99.950000" : 31872,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 5440,
"max" : 126347,
"mean" : 8303.281079,
"stddev" : 2384.467555,
"N" : 3040076
},
"bw_min" : 216236,
"bw_max" : 218096,
"bw_agg" : 100.000000,
"bw_mean" : 217165.357143,
"bw_dev" : 551.447969,
"bw_samples" : 14,
"iops_min" : 432472,
"iops_max" : 436190,
"iops_mean" : 434330.785714,
"iops_stddev" : 1102.928562,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 39.585714,
"sys_cpu" : 60.385714,
"ctx" : 14,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 31.062050,
"4" : 68.935220,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 96.805800,
"20" : 2.581681,
"50" : 0.566894,
"100" : 0.036775,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=9659: Wed Oct 26 15:21:06 2022
read: IOPS=434k, BW=212MiB/s (222MB/s)(1484MiB/7001msec)
slat (nsec): min=1643, max=40897, avg=1833.89, stdev=124.14
clat (nsec): min=290, max=124533, avg=6469.39, stdev=2382.21
lat (usec): min=5, max=126, avg= 8.30, stdev= 2.38
clat percentiles (nsec):
| 1.00th=[ 4448], 5.00th=[ 4448], 10.00th=[ 4576], 20.00th=[ 4704],
| 30.00th=[ 6432], 40.00th=[ 6624], 50.00th=[ 6688], 60.00th=[ 6752],
| 70.00th=[ 6816], 80.00th=[ 6816], 90.00th=[ 6944], 95.00th=[ 7200],
| 99.00th=[17536], 99.50th=[20352], 99.90th=[25728], 99.95th=[31872],
| 99.99th=[66048]
bw ( KiB/s): min=216236, max=218096, per=100.00%, avg=217165.36, stdev=551.45, samples=14
iops : min=432472, max=436190, avg=434330.79, stdev=1102.93, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=96.81%, 20=2.58%, 50=0.57%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=39.59%, sys=60.39%, ctx=14, majf=0, minf=0
IO depths : 1=0.1%, 2=31.1%, 4=68.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3040073,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=212MiB/s (222MB/s), 212MiB/s-212MiB/s (222MB/s-222MB/s), io=1484MiB (1557MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-bdev_aio_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790481,
"timestamp_ms" : 1666790481430,
"time" : "Wed Oct 26 15:21:21 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1547431936,
"io_kbytes" : 1511164,
"bw_bytes" : 221030129,
"bw" : 215849,
"iops" : 431699.042994,
"runtime" : 7001,
"total_ios" : 3022325,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1643,
"max" : 40075,
"mean" : 1835.594190,
"stddev" : 126.051482,
"N" : 3022325
},
"clat_ns" : {
"min" : 300,
"max" : 120215,
"mean" : 6500.137739,
"stddev" : 2390.829741,
"N" : 3022328,
"percentile" : {
"1.000000" : 4448,
"5.000000" : 4512,
"10.000000" : 4576,
"20.000000" : 4704,
"30.000000" : 6432,
"40.000000" : 6688,
"50.000000" : 6752,
"60.000000" : 6752,
"70.000000" : 6816,
"80.000000" : 6880,
"90.000000" : 7072,
"95.000000" : 7264,
"99.000000" : 17536,
"99.500000" : 20608,
"99.900000" : 25984,
"99.950000" : 32128,
"99.990000" : 64768
}
},
"lat_ns" : {
"min" : 5340,
"max" : 122139,
"mean" : 8335.731884,
"stddev" : 2394.300500,
"N" : 3022328
},
"bw_min" : 215090,
"bw_max" : 216619,
"bw_agg" : 100.000000,
"bw_mean" : 215940.769231,
"bw_dev" : 480.123969,
"bw_samples" : 13,
"iops_min" : 430180,
"iops_max" : 433238,
"iops_mean" : 431881.461538,
"iops_stddev" : 960.297056,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 40.442857,
"sys_cpu" : 59.528571,
"ctx" : 15,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 31.311093,
"4" : 68.686789,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.840710,
"20" : 2.537020,
"50" : 0.573764,
"100" : 0.037653,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=9716: Wed Oct 26 15:21:21 2022
read: IOPS=432k, BW=211MiB/s (221MB/s)(1476MiB/7001msec)
slat (nsec): min=1643, max=40075, avg=1835.59, stdev=126.05
clat (nsec): min=300, max=120215, avg=6500.14, stdev=2390.83
lat (usec): min=5, max=122, avg= 8.34, stdev= 2.39
clat percentiles (nsec):
| 1.00th=[ 4448], 5.00th=[ 4512], 10.00th=[ 4576], 20.00th=[ 4704],
| 30.00th=[ 6432], 40.00th=[ 6688], 50.00th=[ 6752], 60.00th=[ 6752],
| 70.00th=[ 6816], 80.00th=[ 6880], 90.00th=[ 7072], 95.00th=[ 7264],
| 99.00th=[17536], 99.50th=[20608], 99.90th=[25984], 99.95th=[32128],
| 99.99th=[64768]
bw ( KiB/s): min=215090, max=216619, per=100.00%, avg=215940.77, stdev=480.12, samples=13
iops : min=430180, max=433238, avg=431881.46, stdev=960.30, samples=13
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=96.84%, 20=2.54%, 50=0.57%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=40.44%, sys=59.53%, ctx=15, majf=0, minf=0
IO depths : 1=0.1%, 2=31.3%, 4=68.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3022325,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=211MiB/s (221MB/s), 211MiB/s-211MiB/s (221MB/s-221MB/s), io=1476MiB (1547MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-reference_0.txt': No such file or directory
$ rm /tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790496,
"timestamp_ms" : 1666790496711,
"time" : "Wed Oct 26 15:21:36 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1444080640,
"io_kbytes" : 1410235,
"bw_bytes" : 206267767,
"bw" : 201433,
"iops" : 402866.161977,
"runtime" : 7001,
"total_ios" : 2820466,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1673,
"max" : 34555,
"mean" : 1859.775083,
"stddev" : 123.879869,
"N" : 2820467
},
"clat_ns" : {
"min" : 701,
"max" : 115096,
"mean" : 6746.611971,
"stddev" : 2343.713145,
"N" : 2820466,
"percentile" : {
"1.000000" : 5216,
"5.000000" : 5280,
"10.000000" : 5280,
"20.000000" : 5344,
"30.000000" : 5408,
"40.000000" : 5472,
"50.000000" : 7264,
"60.000000" : 7328,
"70.000000" : 7392,
"80.000000" : 7456,
"90.000000" : 7584,
"95.000000" : 7968,
"99.000000" : 17536,
"99.500000" : 20608,
"99.900000" : 26240,
"99.950000" : 32128,
"99.990000" : 63744
}
},
"lat_ns" : {
"min" : 5801,
"max" : 116919,
"mean" : 8606.387073,
"stddev" : 2346.267108,
"N" : 2820466
},
"bw_min" : 200600,
"bw_max" : 202229,
"bw_agg" : 100.000000,
"bw_mean" : 201469.857143,
"bw_dev" : 500.196093,
"bw_samples" : 14,
"iops_min" : 401200,
"iops_max" : 404458,
"iops_mean" : 402939.714286,
"iops_stddev" : 1000.392187,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 37.757143,
"sys_cpu" : 62.214286,
"ctx" : 14,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 49.370813,
"4" : 50.626563,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.743020,
"20" : 2.692959,
"50" : 0.519382,
"100" : 0.038150,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=9774: Wed Oct 26 15:21:36 2022
read: IOPS=403k, BW=197MiB/s (206MB/s)(1377MiB/7001msec)
slat (nsec): min=1673, max=34555, avg=1859.78, stdev=123.88
clat (nsec): min=701, max=115096, avg=6746.61, stdev=2343.71
lat (usec): min=5, max=116, avg= 8.61, stdev= 2.35
clat percentiles (nsec):
| 1.00th=[ 5216], 5.00th=[ 5280], 10.00th=[ 5280], 20.00th=[ 5344],
| 30.00th=[ 5408], 40.00th=[ 5472], 50.00th=[ 7264], 60.00th=[ 7328],
| 70.00th=[ 7392], 80.00th=[ 7456], 90.00th=[ 7584], 95.00th=[ 7968],
| 99.00th=[17536], 99.50th=[20608], 99.90th=[26240], 99.95th=[32128],
| 99.99th=[63744]
bw ( KiB/s): min=200600, max=202229, per=100.00%, avg=201469.86, stdev=500.20, samples=14
iops : min=401200, max=404458, avg=402939.71, stdev=1000.39, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 4=0.01%, 10=96.74%, 20=2.69%, 50=0.52%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=37.76%, sys=62.21%, ctx=14, majf=0, minf=0
IO depths : 1=0.1%, 2=49.4%, 4=50.6%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2820466,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=197MiB/s (206MB/s), 197MiB/s-197MiB/s (206MB/s-206MB/s), io=1377MiB (1444MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790512,
"timestamp_ms" : 1666790512099,
"time" : "Wed Oct 26 15:21:52 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1448945664,
"io_kbytes" : 1414986,
"bw_bytes" : 206962671,
"bw" : 202111,
"iops" : 404223.539494,
"runtime" : 7001,
"total_ios" : 2829969,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1653,
"max" : 22372,
"mean" : 1839.170135,
"stddev" : 121.423768,
"N" : 2829970
},
"clat_ns" : {
"min" : 691,
"max" : 120626,
"mean" : 6717.754872,
"stddev" : 2347.788874,
"N" : 2829972,
"percentile" : {
"1.000000" : 5152,
"5.000000" : 5280,
"10.000000" : 5280,
"20.000000" : 5280,
"30.000000" : 5344,
"40.000000" : 5408,
"50.000000" : 7264,
"60.000000" : 7328,
"70.000000" : 7328,
"80.000000" : 7392,
"90.000000" : 7584,
"95.000000" : 7968,
"99.000000" : 17536,
"99.500000" : 20608,
"99.900000" : 26240,
"99.950000" : 32128,
"99.990000" : 64256
}
},
"lat_ns" : {
"min" : 5520,
"max" : 122500,
"mean" : 8556.924986,
"stddev" : 2350.299146,
"N" : 2829972
},
"bw_min" : 201099,
"bw_max" : 203576,
"bw_agg" : 100.000000,
"bw_mean" : 202148.357143,
"bw_dev" : 804.850259,
"bw_samples" : 14,
"iops_min" : 402198,
"iops_max" : 407152,
"iops_mean" : 404296.714286,
"iops_stddev" : 1609.700518,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 39.814286,
"sys_cpu" : 60.157143,
"ctx" : 12,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 49.212165,
"4" : 50.785291,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.734523,
"20" : 2.704376,
"50" : 0.516825,
"100" : 0.037209,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=9831: Wed Oct 26 15:21:52 2022
read: IOPS=404k, BW=197MiB/s (207MB/s)(1382MiB/7001msec)
slat (nsec): min=1653, max=22372, avg=1839.17, stdev=121.42
clat (nsec): min=691, max=120626, avg=6717.75, stdev=2347.79
lat (usec): min=5, max=122, avg= 8.56, stdev= 2.35
clat percentiles (nsec):
| 1.00th=[ 5152], 5.00th=[ 5280], 10.00th=[ 5280], 20.00th=[ 5280],
| 30.00th=[ 5344], 40.00th=[ 5408], 50.00th=[ 7264], 60.00th=[ 7328],
| 70.00th=[ 7328], 80.00th=[ 7392], 90.00th=[ 7584], 95.00th=[ 7968],
| 99.00th=[17536], 99.50th=[20608], 99.90th=[26240], 99.95th=[32128],
| 99.99th=[64256]
bw ( KiB/s): min=201099, max=203576, per=100.00%, avg=202148.36, stdev=804.85, samples=14
iops : min=402198, max=407152, avg=404296.71, stdev=1609.70, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 4=0.01%, 10=96.73%, 20=2.70%, 50=0.52%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=39.81%, sys=60.16%, ctx=12, majf=0, minf=0
IO depths : 1=0.1%, 2=49.2%, 4=50.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2829969,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s), io=1382MiB (1449MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790527,
"timestamp_ms" : 1666790527394,
"time" : "Wed Oct 26 15:22:07 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1440870912,
"io_kbytes" : 1407100,
"bw_bytes" : 205809300,
"bw" : 200985,
"iops" : 401970.718469,
"runtime" : 7001,
"total_ios" : 2814197,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1673,
"max" : 45495,
"mean" : 1855.646163,
"stddev" : 123.276425,
"N" : 2814198
},
"clat_ns" : {
"min" : 682,
"max" : 118602,
"mean" : 6813.198220,
"stddev" : 2388.471879,
"N" : 2814197,
"percentile" : {
"1.000000" : 5216,
"5.000000" : 5280,
"10.000000" : 5280,
"20.000000" : 5344,
"30.000000" : 5344,
"40.000000" : 5472,
"50.000000" : 7264,
"60.000000" : 7328,
"70.000000" : 7392,
"80.000000" : 7456,
"90.000000" : 7904,
"95.000000" : 8096,
"99.000000" : 17536,
"99.500000" : 21376,
"99.900000" : 26496,
"99.950000" : 32128,
"99.990000" : 65280
}
},
"lat_ns" : {
"min" : 5480,
"max" : 120416,
"mean" : 8668.844394,
"stddev" : 2391.963514,
"N" : 2814197
},
"bw_min" : 200134,
"bw_max" : 201911,
"bw_agg" : 100.000000,
"bw_mean" : 201021.500000,
"bw_dev" : 640.262897,
"bw_samples" : 14,
"iops_min" : 400268,
"iops_max" : 403823,
"iops_mean" : 402043.071429,
"iops_stddev" : 1280.632684,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 40.957143,
"sys_cpu" : 59.014286,
"ctx" : 20,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 47.461923,
"4" : 52.534986,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 96.760248,
"20" : 2.542715,
"50" : 0.653970,
"100" : 0.034681,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=9890: Wed Oct 26 15:22:07 2022
read: IOPS=402k, BW=196MiB/s (206MB/s)(1374MiB/7001msec)
slat (nsec): min=1673, max=45495, avg=1855.65, stdev=123.28
clat (nsec): min=682, max=118602, avg=6813.20, stdev=2388.47
lat (usec): min=5, max=120, avg= 8.67, stdev= 2.39
clat percentiles (nsec):
| 1.00th=[ 5216], 5.00th=[ 5280], 10.00th=[ 5280], 20.00th=[ 5344],
| 30.00th=[ 5344], 40.00th=[ 5472], 50.00th=[ 7264], 60.00th=[ 7328],
| 70.00th=[ 7392], 80.00th=[ 7456], 90.00th=[ 7904], 95.00th=[ 8096],
| 99.00th=[17536], 99.50th=[21376], 99.90th=[26496], 99.95th=[32128],
| 99.99th=[65280]
bw ( KiB/s): min=200134, max=201911, per=100.00%, avg=201021.50, stdev=640.26, samples=14
iops : min=400268, max=403823, avg=402043.07, stdev=1280.63, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=96.76%, 20=2.54%, 50=0.65%
lat (usec) : 100=0.03%, 250=0.01%
cpu : usr=40.96%, sys=59.01%, ctx=20, majf=0, minf=0
IO depths : 1=0.1%, 2=47.5%, 4=52.5%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2814197,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=196MiB/s (206MB/s), 196MiB/s-196MiB/s (206MB/s-206MB/s), io=1374MiB (1441MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-reference_0.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790541,
"timestamp_ms" : 1666790541496,
"time" : "Wed Oct 26 15:22:21 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1465430528,
"io_kbytes" : 1431084,
"bw_bytes" : 209317315,
"bw" : 204411,
"iops" : 408822.453935,
"runtime" : 7001,
"total_ios" : 2862166,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1502,
"max" : 21721,
"mean" : 1667.778486,
"stddev" : 106.823115,
"N" : 2862166
},
"clat_ns" : {
"min" : 511,
"max" : 149921,
"mean" : 7891.621895,
"stddev" : 2696.853545,
"N" : 2862169,
"percentile" : {
"1.000000" : 7264,
"5.000000" : 7328,
"10.000000" : 7328,
"20.000000" : 7328,
"30.000000" : 7392,
"40.000000" : 7392,
"50.000000" : 7392,
"60.000000" : 7456,
"70.000000" : 7520,
"80.000000" : 7648,
"90.000000" : 7776,
"95.000000" : 7840,
"99.000000" : 22656,
"99.500000" : 27264,
"99.900000" : 33536,
"99.950000" : 38656,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 6402,
"max" : 151595,
"mean" : 9559.400362,
"stddev" : 2700.206986,
"N" : 2862169
},
"bw_min" : 203864,
"bw_max" : 205193,
"bw_agg" : 100.000000,
"bw_mean" : 204511.357143,
"bw_dev" : 307.759798,
"bw_samples" : 14,
"iops_min" : 407728,
"iops_max" : 410386,
"iops_mean" : 409022.714286,
"iops_stddev" : 615.561711,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 27.228571,
"sys_cpu" : 69.314286,
"ctx" : 25260,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.581889,
"20" : 2.139429,
"50" : 1.239236,
"100" : 0.038922,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=9937: Wed Oct 26 15:22:21 2022
read: IOPS=409k, BW=200MiB/s (209MB/s)(1398MiB/7001msec)
slat (nsec): min=1502, max=21721, avg=1667.78, stdev=106.82
clat (nsec): min=511, max=149921, avg=7891.62, stdev=2696.85
lat (usec): min=6, max=151, avg= 9.56, stdev= 2.70
clat percentiles (nsec):
| 1.00th=[ 7264], 5.00th=[ 7328], 10.00th=[ 7328], 20.00th=[ 7328],
| 30.00th=[ 7392], 40.00th=[ 7392], 50.00th=[ 7392], 60.00th=[ 7456],
| 70.00th=[ 7520], 80.00th=[ 7648], 90.00th=[ 7776], 95.00th=[ 7840],
| 99.00th=[22656], 99.50th=[27264], 99.90th=[33536], 99.95th=[38656],
| 99.99th=[72192]
bw ( KiB/s): min=203864, max=205193, per=100.00%, avg=204511.36, stdev=307.76, samples=14
iops : min=407728, max=410386, avg=409022.71, stdev=615.56, samples=14
lat (nsec) : 750=0.01%
lat (usec) : 10=96.58%, 20=2.14%, 50=1.24%, 100=0.04%, 250=0.01%
cpu : usr=27.23%, sys=69.31%, ctx=25260, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2862166,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=200MiB/s (209MB/s), 200MiB/s-200MiB/s (209MB/s-209MB/s), io=1398MiB (1465MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790555,
"timestamp_ms" : 1666790555276,
"time" : "Wed Oct 26 15:22:35 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1449910272,
"io_kbytes" : 1415928,
"bw_bytes" : 207100453,
"bw" : 202246,
"iops" : 404492.501071,
"runtime" : 7001,
"total_ios" : 2831852,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1533,
"max" : 21871,
"mean" : 1679.587689,
"stddev" : 109.338587,
"N" : 2831853
},
"clat_ns" : {
"min" : 4919,
"max" : 90730,
"mean" : 7972.974219,
"stddev" : 2662.447268,
"N" : 2831855,
"percentile" : {
"1.000000" : 7392,
"5.000000" : 7392,
"10.000000" : 7456,
"20.000000" : 7456,
"30.000000" : 7456,
"40.000000" : 7520,
"50.000000" : 7520,
"60.000000" : 7520,
"70.000000" : 7584,
"80.000000" : 7648,
"90.000000" : 7840,
"95.000000" : 7968,
"99.000000" : 22400,
"99.500000" : 27264,
"99.900000" : 33024,
"99.950000" : 36608,
"99.990000" : 71168
}
},
"lat_ns" : {
"min" : 6472,
"max" : 92383,
"mean" : 9652.561871,
"stddev" : 2665.381488,
"N" : 2831855
},
"bw_min" : 201817,
"bw_max" : 203505,
"bw_agg" : 100.000000,
"bw_mean" : 202362.923077,
"bw_dev" : 461.921072,
"bw_samples" : 13,
"iops_min" : 403634,
"iops_max" : 407010,
"iops_mean" : 404726.153846,
"iops_stddev" : 923.775843,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 32.457143,
"sys_cpu" : 64.142857,
"ctx" : 25084,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.526902,
"20" : 2.143403,
"50" : 1.292864,
"100" : 0.036937,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=9985: Wed Oct 26 15:22:35 2022
read: IOPS=404k, BW=198MiB/s (207MB/s)(1383MiB/7001msec)
slat (nsec): min=1533, max=21871, avg=1679.59, stdev=109.34
clat (nsec): min=4919, max=90730, avg=7972.97, stdev=2662.45
lat (nsec): min=6472, max=92383, avg=9652.56, stdev=2665.38
clat percentiles (nsec):
| 1.00th=[ 7392], 5.00th=[ 7392], 10.00th=[ 7456], 20.00th=[ 7456],
| 30.00th=[ 7456], 40.00th=[ 7520], 50.00th=[ 7520], 60.00th=[ 7520],
| 70.00th=[ 7584], 80.00th=[ 7648], 90.00th=[ 7840], 95.00th=[ 7968],
| 99.00th=[22400], 99.50th=[27264], 99.90th=[33024], 99.95th=[36608],
| 99.99th=[71168]
bw ( KiB/s): min=201817, max=203505, per=100.00%, avg=202362.92, stdev=461.92, samples=13
iops : min=403634, max=407010, avg=404726.15, stdev=923.78, samples=13
lat (usec) : 10=96.53%, 20=2.14%, 50=1.29%, 100=0.04%
cpu : usr=32.46%, sys=64.14%, ctx=25084, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2831852,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=198MiB/s (207MB/s), 198MiB/s-198MiB/s (207MB/s-207MB/s), io=1383MiB (1450MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790570,
"timestamp_ms" : 1666790570033,
"time" : "Wed Oct 26 15:22:50 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1468516864,
"io_kbytes" : 1434098,
"bw_bytes" : 209758157,
"bw" : 204841,
"iops" : 409683.330953,
"runtime" : 7001,
"total_ios" : 2868193,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1513,
"max" : 21661,
"mean" : 1664.947021,
"stddev" : 111.275646,
"N" : 2868194
},
"clat_ns" : {
"min" : 4869,
"max" : 125125,
"mean" : 7874.002652,
"stddev" : 2616.785369,
"N" : 2868196,
"percentile" : {
"1.000000" : 7264,
"5.000000" : 7328,
"10.000000" : 7328,
"20.000000" : 7392,
"30.000000" : 7392,
"40.000000" : 7392,
"50.000000" : 7392,
"60.000000" : 7456,
"70.000000" : 7456,
"80.000000" : 7520,
"90.000000" : 7712,
"95.000000" : 7840,
"99.000000" : 21888,
"99.500000" : 25984,
"99.900000" : 33024,
"99.950000" : 36608,
"99.990000" : 70144
}
},
"lat_ns" : {
"min" : 6412,
"max" : 126818,
"mean" : 9538.949647,
"stddev" : 2618.618149,
"N" : 2868196
},
"bw_min" : 203662,
"bw_max" : 205924,
"bw_agg" : 100.000000,
"bw_mean" : 204950.642857,
"bw_dev" : 680.498189,
"bw_samples" : 14,
"iops_min" : 407324,
"iops_max" : 411848,
"iops_mean" : 409901.571429,
"iops_stddev" : 1360.938913,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 32.400000,
"sys_cpu" : 64.171429,
"ctx" : 25312,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.547025,
"20" : 2.215506,
"50" : 1.198664,
"100" : 0.038526,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=10034: Wed Oct 26 15:22:50 2022
read: IOPS=410k, BW=200MiB/s (210MB/s)(1400MiB/7001msec)
slat (nsec): min=1513, max=21661, avg=1664.95, stdev=111.28
clat (usec): min=4, max=125, avg= 7.87, stdev= 2.62
lat (usec): min=6, max=126, avg= 9.54, stdev= 2.62
clat percentiles (nsec):
| 1.00th=[ 7264], 5.00th=[ 7328], 10.00th=[ 7328], 20.00th=[ 7392],
| 30.00th=[ 7392], 40.00th=[ 7392], 50.00th=[ 7392], 60.00th=[ 7456],
| 70.00th=[ 7456], 80.00th=[ 7520], 90.00th=[ 7712], 95.00th=[ 7840],
| 99.00th=[21888], 99.50th=[25984], 99.90th=[33024], 99.95th=[36608],
| 99.99th=[70144]
bw ( KiB/s): min=203662, max=205924, per=100.00%, avg=204950.64, stdev=680.50, samples=14
iops : min=407324, max=411848, avg=409901.57, stdev=1360.94, samples=14
lat (usec) : 10=96.55%, 20=2.22%, 50=1.20%, 100=0.04%, 250=0.01%
cpu : usr=32.40%, sys=64.17%, ctx=25312, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2868193,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=200MiB/s (210MB/s), 200MiB/s-200MiB/s (210MB/s-210MB/s), io=1400MiB (1469MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789277,
"timestamp_ms" : 1666789277197,
"time" : "Wed Oct 26 15:01:17 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 392608768,
"io_kbytes" : 383407,
"bw_bytes" : 56078955,
"bw" : 54764,
"iops" : 109529.067276,
"runtime" : 7001,
"total_ios" : 766813,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1443,
"max" : 17152,
"mean" : 1576.314540,
"stddev" : 85.440023,
"N" : 766814
},
"clat_ns" : {
"min" : 161,
"max" : 78006,
"mean" : 7235.024115,
"stddev" : 971.994924,
"N" : 766813,
"percentile" : {
"1.000000" : 6752,
"5.000000" : 6944,
"10.000000" : 7008,
"20.000000" : 7008,
"30.000000" : 7072,
"40.000000" : 7072,
"50.000000" : 7136,
"60.000000" : 7200,
"70.000000" : 7200,
"80.000000" : 7264,
"90.000000" : 7328,
"95.000000" : 7456,
"99.000000" : 10304,
"99.500000" : 13504,
"99.900000" : 15552,
"99.950000" : 15808,
"99.990000" : 58112
}
},
"lat_ns" : {
"min" : 7444,
"max" : 79509,
"mean" : 8811.338725,
"stddev" : 977.342601,
"N" : 766813
},
"bw_min" : 54549,
"bw_max" : 55065,
"bw_agg" : 100.000000,
"bw_mean" : 54801.642857,
"bw_dev" : 147.506249,
"bw_samples" : 14,
"iops_min" : 109098,
"iops_max" : 110130,
"iops_mean" : 109603.500000,
"iops_stddev" : 295.184883,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 9.400000,
"sys_cpu" : 34.500000,
"ctx" : 766814,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 98.745718,
"20" : 1.234330,
"50" : 0.010000,
"100" : 0.013954,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 1099741,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 5523,
"write_ticks" : 0,
"in_queue" : 5523,
"util" : 99.108381
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=5391: Wed Oct 26 15:01:17 2022
read: IOPS=110k, BW=53.5MiB/s (56.1MB/s)(374MiB/7001msec)
slat (nsec): min=1443, max=17152, avg=1576.31, stdev=85.44
clat (nsec): min=161, max=78006, avg=7235.02, stdev=971.99
lat (nsec): min=7444, max=79509, avg=8811.34, stdev=977.34
clat percentiles (nsec):
| 1.00th=[ 6752], 5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7008],
| 30.00th=[ 7072], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7200],
| 70.00th=[ 7200], 80.00th=[ 7264], 90.00th=[ 7328], 95.00th=[ 7456],
| 99.00th=[10304], 99.50th=[13504], 99.90th=[15552], 99.95th=[15808],
| 99.99th=[58112]
bw ( KiB/s): min=54549, max=55065, per=100.00%, avg=54801.64, stdev=147.51, samples=14
iops : min=109098, max=110130, avg=109603.50, stdev=295.18, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=98.75%, 20=1.23%, 50=0.01%, 100=0.01%
cpu : usr=9.40%, sys=34.50%, ctx=766814, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=766813,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=53.5MiB/s (56.1MB/s), 53.5MiB/s-53.5MiB/s (56.1MB/s-56.1MB/s), io=374MiB (393MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=1099741/0, merge=0/0, ticks=5523/0, in_queue=5523, util=99.11%
$ cat /tmp/fio-output_512_8_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790583,
"timestamp_ms" : 1666790583831,
"time" : "Wed Oct 26 15:23:03 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1613466624,
"io_kbytes" : 1575651,
"bw_bytes" : 230462308,
"bw" : 225060,
"iops" : 450120.554207,
"runtime" : 7001,
"total_ios" : 3151294,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1172,
"max" : 51907,
"mean" : 1784.391308,
"stddev" : 386.919507,
"N" : 3151295
},
"clat_ns" : {
"min" : 5240,
"max" : 113272,
"mean" : 15718.233979,
"stddev" : 3820.322070,
"N" : 3151301,
"percentile" : {
"1.000000" : 12224,
"5.000000" : 12736,
"10.000000" : 13120,
"20.000000" : 14272,
"30.000000" : 14784,
"40.000000" : 14912,
"50.000000" : 15040,
"60.000000" : 15424,
"70.000000" : 15680,
"80.000000" : 16320,
"90.000000" : 17280,
"95.000000" : 18048,
"99.000000" : 36608,
"99.500000" : 39168,
"99.900000" : 48384,
"99.950000" : 64256,
"99.990000" : 79360
}
},
"lat_ns" : {
"min" : 6453,
"max" : 114505,
"mean" : 17502.625444,
"stddev" : 3781.638897,
"N" : 3151301
},
"bw_min" : 224723,
"bw_max" : 225795,
"bw_agg" : 100.000000,
"bw_mean" : 225187.928571,
"bw_dev" : 301.062085,
"bw_samples" : 14,
"iops_min" : 449446,
"iops_max" : 451591,
"iops_mean" : 450376.071429,
"iops_stddev" : 602.094097,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 16.871429,
"sys_cpu" : 79.900000,
"ctx" : 15021,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.044109,
"20" : 96.321670,
"50" : 3.541847,
"100" : 0.092248,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=10082: Wed Oct 26 15:23:03 2022
read: IOPS=450k, BW=220MiB/s (230MB/s)(1539MiB/7001msec)
slat (nsec): min=1172, max=51907, avg=1784.39, stdev=386.92
clat (usec): min=5, max=113, avg=15.72, stdev= 3.82
lat (usec): min=6, max=114, avg=17.50, stdev= 3.78
clat percentiles (nsec):
| 1.00th=[12224], 5.00th=[12736], 10.00th=[13120], 20.00th=[14272],
| 30.00th=[14784], 40.00th=[14912], 50.00th=[15040], 60.00th=[15424],
| 70.00th=[15680], 80.00th=[16320], 90.00th=[17280], 95.00th=[18048],
| 99.00th=[36608], 99.50th=[39168], 99.90th=[48384], 99.95th=[64256],
| 99.99th=[79360]
bw ( KiB/s): min=224723, max=225795, per=100.00%, avg=225187.93, stdev=301.06, samples=14
iops : min=449446, max=451591, avg=450376.07, stdev=602.09, samples=14
lat (usec) : 10=0.04%, 20=96.32%, 50=3.54%, 100=0.09%, 250=0.01%
cpu : usr=16.87%, sys=79.90%, ctx=15021, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3151294,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=220MiB/s (230MB/s), 220MiB/s-220MiB/s (230MB/s-230MB/s), io=1539MiB (1613MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790597,
"timestamp_ms" : 1666790597635,
"time" : "Wed Oct 26 15:23:17 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1596023808,
"io_kbytes" : 1558617,
"bw_bytes" : 227970833,
"bw" : 222627,
"iops" : 445254.392230,
"runtime" : 7001,
"total_ios" : 3117226,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1192,
"max" : 26660,
"mean" : 1791.133760,
"stddev" : 379.567079,
"N" : 3117227
},
"clat_ns" : {
"min" : 1963,
"max" : 125385,
"mean" : 15887.595758,
"stddev" : 3772.244360,
"N" : 3117233,
"percentile" : {
"1.000000" : 12224,
"5.000000" : 12864,
"10.000000" : 13248,
"20.000000" : 14528,
"30.000000" : 14912,
"40.000000" : 15168,
"50.000000" : 15296,
"60.000000" : 15680,
"70.000000" : 15936,
"80.000000" : 16512,
"90.000000" : 17536,
"95.000000" : 18048,
"99.000000" : 36608,
"99.500000" : 38656,
"99.900000" : 47872,
"99.950000" : 63744,
"99.990000" : 79360
}
},
"lat_ns" : {
"min" : 6492,
"max" : 126838,
"mean" : 17678.729535,
"stddev" : 3741.390408,
"N" : 3117233
},
"bw_min" : 221913,
"bw_max" : 223653,
"bw_agg" : 100.000000,
"bw_mean" : 222667.071429,
"bw_dev" : 489.091224,
"bw_samples" : 14,
"iops_min" : 443826,
"iops_max" : 447306,
"iops_mean" : 445334.142857,
"iops_stddev" : 978.182448,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.685714,
"sys_cpu" : 79.200000,
"ctx" : 14928,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.050077,
"20" : 96.374341,
"50" : 3.485727,
"100" : 0.089374,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=10128: Wed Oct 26 15:23:17 2022
read: IOPS=445k, BW=217MiB/s (228MB/s)(1522MiB/7001msec)
slat (nsec): min=1192, max=26660, avg=1791.13, stdev=379.57
clat (nsec): min=1963, max=125385, avg=15887.60, stdev=3772.24
lat (usec): min=6, max=126, avg=17.68, stdev= 3.74
clat percentiles (nsec):
| 1.00th=[12224], 5.00th=[12864], 10.00th=[13248], 20.00th=[14528],
| 30.00th=[14912], 40.00th=[15168], 50.00th=[15296], 60.00th=[15680],
| 70.00th=[15936], 80.00th=[16512], 90.00th=[17536], 95.00th=[18048],
| 99.00th=[36608], 99.50th=[38656], 99.90th=[47872], 99.95th=[63744],
| 99.99th=[79360]
bw ( KiB/s): min=221913, max=223653, per=100.00%, avg=222667.07, stdev=489.09, samples=14
iops : min=443826, max=447306, avg=445334.14, stdev=978.18, samples=14
lat (usec) : 2=0.01%, 4=0.01%, 10=0.05%, 20=96.37%, 50=3.49%
lat (usec) : 100=0.09%, 250=0.01%
cpu : usr=17.69%, sys=79.20%, ctx=14928, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3117226,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=217MiB/s (228MB/s), 217MiB/s-217MiB/s (228MB/s-228MB/s), io=1522MiB (1596MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790611,
"timestamp_ms" : 1666790611370,
"time" : "Wed Oct 26 15:23:31 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1604506112,
"io_kbytes" : 1566900,
"bw_bytes" : 229182418,
"bw" : 223810,
"iops" : 447620.768462,
"runtime" : 7001,
"total_ios" : 3133793,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1162,
"max" : 23243,
"mean" : 1790.342289,
"stddev" : 400.121691,
"N" : 3133794
},
"clat_ns" : {
"min" : 1874,
"max" : 130384,
"mean" : 15806.641568,
"stddev" : 3805.133567,
"N" : 3133800,
"percentile" : {
"1.000000" : 12096,
"5.000000" : 12736,
"10.000000" : 13248,
"20.000000" : 14400,
"30.000000" : 14784,
"40.000000" : 15040,
"50.000000" : 15168,
"60.000000" : 15552,
"70.000000" : 15808,
"80.000000" : 16512,
"90.000000" : 17280,
"95.000000" : 18048,
"99.000000" : 36608,
"99.500000" : 38656,
"99.900000" : 48384,
"99.950000" : 64256,
"99.990000" : 79360
}
},
"lat_ns" : {
"min" : 6532,
"max" : 131657,
"mean" : 17596.983862,
"stddev" : 3768.535898,
"N" : 3133800
},
"bw_min" : 222819,
"bw_max" : 225676,
"bw_agg" : 100.000000,
"bw_mean" : 223915.384615,
"bw_dev" : 891.737867,
"bw_samples" : 13,
"iops_min" : 445638,
"iops_max" : 451352,
"iops_mean" : 447830.846154,
"iops_stddev" : 1783.386285,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.214286,
"sys_cpu" : 79.585714,
"ctx" : 14974,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.049142,
"20" : 96.312360,
"50" : 3.546022,
"100" : 0.092412,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=10175: Wed Oct 26 15:23:31 2022
read: IOPS=448k, BW=219MiB/s (229MB/s)(1530MiB/7001msec)
slat (nsec): min=1162, max=23243, avg=1790.34, stdev=400.12
clat (nsec): min=1874, max=130384, avg=15806.64, stdev=3805.13
lat (usec): min=6, max=131, avg=17.60, stdev= 3.77
clat percentiles (nsec):
| 1.00th=[12096], 5.00th=[12736], 10.00th=[13248], 20.00th=[14400],
| 30.00th=[14784], 40.00th=[15040], 50.00th=[15168], 60.00th=[15552],
| 70.00th=[15808], 80.00th=[16512], 90.00th=[17280], 95.00th=[18048],
| 99.00th=[36608], 99.50th=[38656], 99.90th=[48384], 99.95th=[64256],
| 99.99th=[79360]
bw ( KiB/s): min=222819, max=225676, per=100.00%, avg=223915.38, stdev=891.74, samples=13
iops : min=445638, max=451352, avg=447830.85, stdev=1783.39, samples=13
lat (usec) : 2=0.01%, 10=0.05%, 20=96.31%, 50=3.55%, 100=0.09%
lat (usec) : 250=0.01%
cpu : usr=17.21%, sys=79.59%, ctx=14974, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3133793,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=219MiB/s (229MB/s), 219MiB/s-219MiB/s (229MB/s-229MB/s), io=1530MiB (1605MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790625,
"timestamp_ms" : 1666790625309,
"time" : "Wed Oct 26 15:23:45 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1514349056,
"io_kbytes" : 1478856,
"bw_bytes" : 216335579,
"bw" : 211265,
"iops" : 422529.285714,
"runtime" : 7000,
"total_ios" : 2957705,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1232,
"max" : 23514,
"mean" : 1841.139255,
"stddev" : 389.694801,
"N" : 2957706
},
"clat_ns" : {
"min" : 3677,
"max" : 120536,
"mean" : 16780.929818,
"stddev" : 3696.313221,
"N" : 2957712,
"percentile" : {
"1.000000" : 12608,
"5.000000" : 13248,
"10.000000" : 14400,
"20.000000" : 15296,
"30.000000" : 15808,
"40.000000" : 16192,
"50.000000" : 16512,
"60.000000" : 16768,
"70.000000" : 17024,
"80.000000" : 17536,
"90.000000" : 17792,
"95.000000" : 18560,
"99.000000" : 36608,
"99.500000" : 38656,
"99.900000" : 50432,
"99.950000" : 66048,
"99.990000" : 81408
}
},
"lat_ns" : {
"min" : 6672,
"max" : 121879,
"mean" : 18622.069282,
"stddev" : 3671.329577,
"N" : 2957712
},
"bw_min" : 208536,
"bw_max" : 213329,
"bw_agg" : 100.000000,
"bw_mean" : 211349.428571,
"bw_dev" : 1221.796830,
"bw_samples" : 14,
"iops_min" : 417072,
"iops_max" : 426658,
"iops_mean" : 422699.000000,
"iops_stddev" : 2443.594893,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 20.442857,
"sys_cpu" : 76.714286,
"ctx" : 14856,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999983,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.040910,
"20" : 96.235764,
"50" : 3.622944,
"100" : 0.100111,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=10221: Wed Oct 26 15:23:45 2022
read: IOPS=423k, BW=206MiB/s (216MB/s)(1444MiB/7000msec)
slat (nsec): min=1232, max=23514, avg=1841.14, stdev=389.69
clat (usec): min=3, max=120, avg=16.78, stdev= 3.70
lat (usec): min=6, max=121, avg=18.62, stdev= 3.67
clat percentiles (nsec):
| 1.00th=[12608], 5.00th=[13248], 10.00th=[14400], 20.00th=[15296],
| 30.00th=[15808], 40.00th=[16192], 50.00th=[16512], 60.00th=[16768],
| 70.00th=[17024], 80.00th=[17536], 90.00th=[17792], 95.00th=[18560],
| 99.00th=[36608], 99.50th=[38656], 99.90th=[50432], 99.95th=[66048],
| 99.99th=[81408]
bw ( KiB/s): min=208536, max=213329, per=100.00%, avg=211349.43, stdev=1221.80, samples=14
iops : min=417072, max=426658, avg=422699.00, stdev=2443.59, samples=14
lat (usec) : 4=0.01%, 10=0.04%, 20=96.24%, 50=3.62%, 100=0.10%
lat (usec) : 250=0.01%
cpu : usr=20.44%, sys=76.71%, ctx=14856, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2957705,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=206MiB/s (216MB/s), 206MiB/s-206MiB/s (216MB/s-216MB/s), io=1444MiB (1514MB), run=7000-7000msec
$ rm /tmp/fio-output_512_1_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-reference_1.txt': No such file or directory
$ rm /tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790639,
"timestamp_ms" : 1666790639058,
"time" : "Wed Oct 26 15:23:59 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1536173056,
"io_kbytes" : 1500169,
"bw_bytes" : 219421947,
"bw" : 214279,
"iops" : 428557.491787,
"runtime" : 7001,
"total_ios" : 3000331,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1222,
"max" : 23965,
"mean" : 1803.258623,
"stddev" : 390.548360,
"N" : 3000331
},
"clat_ns" : {
"min" : 4839,
"max" : 128591,
"mean" : 16544.098031,
"stddev" : 3761.336130,
"N" : 3000338,
"percentile" : {
"1.000000" : 12608,
"5.000000" : 13120,
"10.000000" : 13760,
"20.000000" : 15040,
"30.000000" : 15552,
"40.000000" : 15936,
"50.000000" : 16192,
"60.000000" : 16512,
"70.000000" : 16768,
"80.000000" : 17280,
"90.000000" : 17536,
"95.000000" : 18560,
"99.000000" : 37120,
"99.500000" : 39168,
"99.900000" : 51456,
"99.950000" : 67072,
"99.990000" : 81408
}
},
"lat_ns" : {
"min" : 6191,
"max" : 129934,
"mean" : 18347.356140,
"stddev" : 3737.823304,
"N" : 3000338
},
"bw_min" : 211728,
"bw_max" : 216886,
"bw_agg" : 100.000000,
"bw_mean" : 214318.285714,
"bw_dev" : 1642.356301,
"bw_samples" : 14,
"iops_min" : 423456,
"iops_max" : 433772,
"iops_mean" : 428636.857143,
"iops_stddev" : 3284.857914,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 22.057143,
"sys_cpu" : 75.014286,
"ctx" : 14798,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999983,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.043895,
"20" : 96.254080,
"50" : 3.599036,
"100" : 0.101522,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=10267: Wed Oct 26 15:23:59 2022
read: IOPS=429k, BW=209MiB/s (219MB/s)(1465MiB/7001msec)
slat (nsec): min=1222, max=23965, avg=1803.26, stdev=390.55
clat (usec): min=4, max=128, avg=16.54, stdev= 3.76
lat (usec): min=6, max=129, avg=18.35, stdev= 3.74
clat percentiles (nsec):
| 1.00th=[12608], 5.00th=[13120], 10.00th=[13760], 20.00th=[15040],
| 30.00th=[15552], 40.00th=[15936], 50.00th=[16192], 60.00th=[16512],
| 70.00th=[16768], 80.00th=[17280], 90.00th=[17536], 95.00th=[18560],
| 99.00th=[37120], 99.50th=[39168], 99.90th=[51456], 99.95th=[67072],
| 99.99th=[81408]
bw ( KiB/s): min=211728, max=216886, per=100.00%, avg=214318.29, stdev=1642.36, samples=14
iops : min=423456, max=433772, avg=428636.86, stdev=3284.86, samples=14
lat (usec) : 10=0.04%, 20=96.25%, 50=3.60%, 100=0.10%, 250=0.01%
cpu : usr=22.06%, sys=75.01%, ctx=14798, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3000331,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=209MiB/s (219MB/s), 209MiB/s-209MiB/s (219MB/s-219MB/s), io=1465MiB (1536MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790652,
"timestamp_ms" : 1666790652800,
"time" : "Wed Oct 26 15:24:12 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1528946176,
"io_kbytes" : 1493111,
"bw_bytes" : 218389683,
"bw" : 213271,
"iops" : 426541.208399,
"runtime" : 7001,
"total_ios" : 2986215,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1222,
"max" : 37791,
"mean" : 1814.158787,
"stddev" : 388.864751,
"N" : 2986216
},
"clat_ns" : {
"min" : 121,
"max" : 127819,
"mean" : 16622.123058,
"stddev" : 3687.178497,
"N" : 2986222,
"percentile" : {
"1.000000" : 12608,
"5.000000" : 13248,
"10.000000" : 14144,
"20.000000" : 15296,
"30.000000" : 15680,
"40.000000" : 16064,
"50.000000" : 16192,
"60.000000" : 16512,
"70.000000" : 16768,
"80.000000" : 17280,
"90.000000" : 17536,
"95.000000" : 18560,
"99.000000" : 36608,
"99.500000" : 38656,
"99.900000" : 48896,
"99.950000" : 65280,
"99.990000" : 80384
}
},
"lat_ns" : {
"min" : 6693,
"max" : 129142,
"mean" : 18436.281854,
"stddev" : 3660.248095,
"N" : 2986222
},
"bw_min" : 212347,
"bw_max" : 214606,
"bw_agg" : 100.000000,
"bw_mean" : 213309.142857,
"bw_dev" : 669.714733,
"bw_samples" : 14,
"iops_min" : 424694,
"iops_max" : 429212,
"iops_mean" : 426618.285714,
"iops_stddev" : 1339.429467,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 21.257143,
"sys_cpu" : 75.814286,
"ctx" : 14684,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999983,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.037707,
"20" : 96.299061,
"50" : 3.567292,
"100" : 0.095405,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=10313: Wed Oct 26 15:24:12 2022
read: IOPS=427k, BW=208MiB/s (218MB/s)(1458MiB/7001msec)
slat (nsec): min=1222, max=37791, avg=1814.16, stdev=388.86
clat (nsec): min=121, max=127819, avg=16622.12, stdev=3687.18
lat (usec): min=6, max=129, avg=18.44, stdev= 3.66
clat percentiles (nsec):
| 1.00th=[12608], 5.00th=[13248], 10.00th=[14144], 20.00th=[15296],
| 30.00th=[15680], 40.00th=[16064], 50.00th=[16192], 60.00th=[16512],
| 70.00th=[16768], 80.00th=[17280], 90.00th=[17536], 95.00th=[18560],
| 99.00th=[36608], 99.50th=[38656], 99.90th=[48896], 99.95th=[65280],
| 99.99th=[80384]
bw ( KiB/s): min=212347, max=214606, per=100.00%, avg=213309.14, stdev=669.71, samples=14
iops : min=424694, max=429212, avg=426618.29, stdev=1339.43, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 10=0.04%, 20=96.30%, 50=3.57%, 100=0.10%, 250=0.01%
cpu : usr=21.26%, sys=75.81%, ctx=14684, majf=0, minf=1
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2986215,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=208MiB/s (218MB/s), 208MiB/s-208MiB/s (218MB/s-218MB/s), io=1458MiB (1529MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790667,
"timestamp_ms" : 1666790667990,
"time" : "Wed Oct 26 15:24:27 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 2123516928,
"io_kbytes" : 2073747,
"bw_bytes" : 303316230,
"bw" : 296207,
"iops" : 592413.512355,
"runtime" : 7001,
"total_ios" : 4147487,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 190,
"max" : 26219,
"mean" : 278.529163,
"stddev" : 75.839500,
"N" : 4147488
},
"clat_ns" : {
"min" : 691,
"max" : 131106,
"mean" : 12536.847786,
"stddev" : 4087.236657,
"N" : 4147494,
"percentile" : {
"1.000000" : 7008,
"5.000000" : 7904,
"10.000000" : 8640,
"20.000000" : 9664,
"30.000000" : 10688,
"40.000000" : 11456,
"50.000000" : 12096,
"60.000000" : 12608,
"70.000000" : 13504,
"80.000000" : 14656,
"90.000000" : 15936,
"95.000000" : 17280,
"99.000000" : 29568,
"99.500000" : 32128,
"99.900000" : 43776,
"99.950000" : 62720,
"99.990000" : 76288
}
},
"lat_ns" : {
"min" : 5720,
"max" : 131396,
"mean" : 12815.376954,
"stddev" : 4084.439124,
"N" : 4147494
},
"bw_min" : 295554,
"bw_max" : 296921,
"bw_agg" : 100.000000,
"bw_mean" : 296335.857143,
"bw_dev" : 340.016483,
"bw_samples" : 14,
"iops_min" : 591108,
"iops_max" : 593843,
"iops_mean" : 592671.857143,
"iops_stddev" : 680.086969,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 21,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.100000,
"4" : 35.838135,
"8" : 64.137537,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.802206,
"8" : 0.197794,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 23.339482,
"20" : 73.205245,
"50" : 3.369510,
"100" : 0.084557,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=10371: Wed Oct 26 15:24:27 2022
read: IOPS=592k, BW=289MiB/s (303MB/s)(2025MiB/7001msec)
slat (nsec): min=190, max=26219, avg=278.53, stdev=75.84
clat (nsec): min=691, max=131106, avg=12536.85, stdev=4087.24
lat (usec): min=5, max=131, avg=12.82, stdev= 4.08
clat percentiles (nsec):
| 1.00th=[ 7008], 5.00th=[ 7904], 10.00th=[ 8640], 20.00th=[ 9664],
| 30.00th=[10688], 40.00th=[11456], 50.00th=[12096], 60.00th=[12608],
| 70.00th=[13504], 80.00th=[14656], 90.00th=[15936], 95.00th=[17280],
| 99.00th=[29568], 99.50th=[32128], 99.90th=[43776], 99.95th=[62720],
| 99.99th=[76288]
bw ( KiB/s): min=295554, max=296921, per=100.00%, avg=296335.86, stdev=340.02, samples=14
iops : min=591108, max=593843, avg=592671.86, stdev=680.09, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=23.34%, 20=73.21%, 50=3.37%
lat (usec) : 100=0.08%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=21, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=35.8%, 8=64.1%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.8%, 8=0.2%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4147487,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=289MiB/s (303MB/s), 289MiB/s-289MiB/s (303MB/s-303MB/s), io=2025MiB (2124MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-reference_1.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790683,
"timestamp_ms" : 1666790683397,
"time" : "Wed Oct 26 15:24:43 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1576720896,
"io_kbytes" : 1539766,
"bw_bytes" : 225213668,
"bw" : 219935,
"iops" : 439869.447222,
"runtime" : 7001,
"total_ios" : 3079526,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 170,
"max" : 30026,
"mean" : 708.241025,
"stddev" : 279.522443,
"N" : 3079527
},
"clat_ns" : {
"min" : 711,
"max" : 200556,
"mean" : 16166.595282,
"stddev" : 5857.765495,
"N" : 3079533,
"percentile" : {
"1.000000" : 7456,
"5.000000" : 8768,
"10.000000" : 9408,
"20.000000" : 11072,
"30.000000" : 12480,
"40.000000" : 13632,
"50.000000" : 15168,
"60.000000" : 16768,
"70.000000" : 18816,
"80.000000" : 21376,
"90.000000" : 23936,
"95.000000" : 25728,
"99.000000" : 33024,
"99.500000" : 36096,
"99.900000" : 47360,
"99.950000" : 64256,
"99.990000" : 77312
}
},
"lat_ns" : {
"min" : 5370,
"max" : 202119,
"mean" : 16874.835481,
"stddev" : 5927.145293,
"N" : 3079533
},
"bw_min" : 198922,
"bw_max" : 296648,
"bw_agg" : 100.000000,
"bw_mean" : 220001.714286,
"bw_dev" : 36653.514516,
"bw_samples" : 14,
"iops_min" : 397844,
"iops_max" : 593296,
"iops_mean" : 440003.428571,
"iops_stddev" : 73307.214011,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 21,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 3.569933,
"4" : 45.009070,
"8" : 51.419309,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 97.162062,
"8" : 2.837938,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 13.485322,
"20" : 61.459426,
"50" : 24.963355,
"100" : 0.090858,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=10431: Wed Oct 26 15:24:43 2022
read: IOPS=440k, BW=215MiB/s (225MB/s)(1504MiB/7001msec)
slat (nsec): min=170, max=30026, avg=708.24, stdev=279.52
clat (nsec): min=711, max=200556, avg=16166.60, stdev=5857.77
lat (usec): min=5, max=202, avg=16.87, stdev= 5.93
clat percentiles (nsec):
| 1.00th=[ 7456], 5.00th=[ 8768], 10.00th=[ 9408], 20.00th=[11072],
| 30.00th=[12480], 40.00th=[13632], 50.00th=[15168], 60.00th=[16768],
| 70.00th=[18816], 80.00th=[21376], 90.00th=[23936], 95.00th=[25728],
| 99.00th=[33024], 99.50th=[36096], 99.90th=[47360], 99.95th=[64256],
| 99.99th=[77312]
bw ( KiB/s): min=198922, max=296648, per=100.00%, avg=220001.71, stdev=36653.51, samples=14
iops : min=397844, max=593296, avg=440003.43, stdev=73307.21, samples=14
lat (nsec) : 750=0.01%
lat (usec) : 2=0.01%, 10=13.49%, 20=61.46%, 50=24.96%, 100=0.09%
lat (usec) : 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=21, majf=0, minf=0
IO depths : 1=0.1%, 2=3.6%, 4=45.0%, 8=51.4%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=97.2%, 8=2.8%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3079526,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=215MiB/s (225MB/s), 215MiB/s-215MiB/s (225MB/s-225MB/s), io=1504MiB (1577MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790698,
"timestamp_ms" : 1666790698756,
"time" : "Wed Oct 26 15:24:58 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1556664832,
"io_kbytes" : 1520180,
"bw_bytes" : 222348926,
"bw" : 217137,
"iops" : 434274.246536,
"runtime" : 7001,
"total_ios" : 3040354,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 631,
"max" : 14688,
"mean" : 866.610232,
"stddev" : 144.688161,
"N" : 3040354
},
"clat_ns" : {
"min" : 6462,
"max" : 132108,
"mean" : 16360.181632,
"stddev" : 5137.436929,
"N" : 3040360,
"percentile" : {
"1.000000" : 8384,
"5.000000" : 9536,
"10.000000" : 10304,
"20.000000" : 12224,
"30.000000" : 13376,
"40.000000" : 14528,
"50.000000" : 15680,
"60.000000" : 17024,
"70.000000" : 18816,
"80.000000" : 20096,
"90.000000" : 22144,
"95.000000" : 24960,
"99.000000" : 32640,
"99.500000" : 36096,
"99.900000" : 44800,
"99.950000" : 61696,
"99.990000" : 76288
}
},
"lat_ns" : {
"min" : 7364,
"max" : 132779,
"mean" : 17226.791887,
"stddev" : 5119.625326,
"N" : 3040360
},
"bw_min" : 216825,
"bw_max" : 217381,
"bw_agg" : 100.000000,
"bw_mean" : 217176.142857,
"bw_dev" : 164.800045,
"bw_samples" : 14,
"iops_min" : 433650,
"iops_max" : 434762,
"iops_mean" : 434352.357143,
"iops_stddev" : 329.733134,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 16,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 1.185553,
"4" : 45.805258,
"8" : 53.008959,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 98.136979,
"8" : 1.863021,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 8.138000,
"20" : 70.711042,
"50" : 21.071198,
"100" : 0.078215,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=10490: Wed Oct 26 15:24:58 2022
read: IOPS=434k, BW=212MiB/s (222MB/s)(1485MiB/7001msec)
slat (nsec): min=631, max=14688, avg=866.61, stdev=144.69
clat (usec): min=6, max=132, avg=16.36, stdev= 5.14
lat (usec): min=7, max=132, avg=17.23, stdev= 5.12
clat percentiles (nsec):
| 1.00th=[ 8384], 5.00th=[ 9536], 10.00th=[10304], 20.00th=[12224],
| 30.00th=[13376], 40.00th=[14528], 50.00th=[15680], 60.00th=[17024],
| 70.00th=[18816], 80.00th=[20096], 90.00th=[22144], 95.00th=[24960],
| 99.00th=[32640], 99.50th=[36096], 99.90th=[44800], 99.95th=[61696],
| 99.99th=[76288]
bw ( KiB/s): min=216825, max=217381, per=100.00%, avg=217176.14, stdev=164.80, samples=14
iops : min=433650, max=434762, avg=434352.36, stdev=329.73, samples=14
lat (usec) : 10=8.14%, 20=70.71%, 50=21.07%, 100=0.08%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=16, majf=0, minf=0
IO depths : 1=0.1%, 2=1.2%, 4=45.8%, 8=53.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=98.1%, 8=1.9%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3040354,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=212MiB/s (222MB/s), 212MiB/s-212MiB/s (222MB/s-222MB/s), io=1485MiB (1557MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790712,
"timestamp_ms" : 1666790712555,
"time" : "Wed Oct 26 15:25:12 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1552151040,
"io_kbytes" : 1515772,
"bw_bytes" : 221704190,
"bw" : 216507,
"iops" : 433014.855021,
"runtime" : 7001,
"total_ios" : 3031537,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1332,
"max" : 31810,
"mean" : 1862.089115,
"stddev" : 370.306038,
"N" : 3031538
},
"clat_ns" : {
"min" : 2144,
"max" : 123952,
"mean" : 16334.740680,
"stddev" : 3609.143862,
"N" : 3031544,
"percentile" : {
"1.000000" : 12480,
"5.000000" : 12992,
"10.000000" : 13248,
"20.000000" : 14784,
"30.000000" : 15296,
"40.000000" : 15552,
"50.000000" : 15936,
"60.000000" : 16192,
"70.000000" : 16768,
"80.000000" : 17280,
"90.000000" : 17792,
"95.000000" : 18816,
"99.000000" : 32640,
"99.500000" : 35072,
"99.900000" : 51456,
"99.950000" : 67072,
"99.990000" : 78336
}
},
"lat_ns" : {
"min" : 7133,
"max" : 125345,
"mean" : 18196.829907,
"stddev" : 3573.540743,
"N" : 3031544
},
"bw_min" : 215528,
"bw_max" : 217965,
"bw_agg" : 100.000000,
"bw_mean" : 216654.571429,
"bw_dev" : 805.018749,
"bw_samples" : 14,
"iops_min" : 431056,
"iops_max" : 435931,
"iops_mean" : 433309.285714,
"iops_stddev" : 1610.180087,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 16.757143,
"sys_cpu" : 80.328571,
"ctx" : 15022,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999967,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.056506,
"20" : 95.943741,
"50" : 3.889281,
"100" : 0.109647,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 4373344,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 25161,
"write_ticks" : 0,
"in_queue" : 25161,
"util" : 99.108381
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=10537: Wed Oct 26 15:25:12 2022
read: IOPS=433k, BW=211MiB/s (222MB/s)(1480MiB/7001msec)
slat (nsec): min=1332, max=31810, avg=1862.09, stdev=370.31
clat (usec): min=2, max=123, avg=16.33, stdev= 3.61
lat (usec): min=7, max=125, avg=18.20, stdev= 3.57
clat percentiles (nsec):
| 1.00th=[12480], 5.00th=[12992], 10.00th=[13248], 20.00th=[14784],
| 30.00th=[15296], 40.00th=[15552], 50.00th=[15936], 60.00th=[16192],
| 70.00th=[16768], 80.00th=[17280], 90.00th=[17792], 95.00th=[18816],
| 99.00th=[32640], 99.50th=[35072], 99.90th=[51456], 99.95th=[67072],
| 99.99th=[78336]
bw ( KiB/s): min=215528, max=217965, per=100.00%, avg=216654.57, stdev=805.02, samples=14
iops : min=431056, max=435931, avg=433309.29, stdev=1610.18, samples=14
lat (usec) : 4=0.01%, 10=0.06%, 20=95.94%, 50=3.89%, 100=0.11%
lat (usec) : 250=0.01%
cpu : usr=16.76%, sys=80.33%, ctx=15022, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3031537,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=211MiB/s (222MB/s), 211MiB/s-211MiB/s (222MB/s-222MB/s), io=1480MiB (1552MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=4373344/0, merge=0/0, ticks=25161/0, in_queue=25161, util=99.11%
$ rm /tmp/fio-output_512_8_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789291,
"timestamp_ms" : 1666789291136,
"time" : "Wed Oct 26 15:01:31 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 392546816,
"io_kbytes" : 383346,
"bw_bytes" : 56070106,
"bw" : 54755,
"iops" : 109511.784031,
"runtime" : 7001,
"total_ios" : 766692,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1452,
"max" : 7204,
"mean" : 1551.774812,
"stddev" : 73.369744,
"N" : 766693
},
"clat_ns" : {
"min" : 1042,
"max" : 78077,
"mean" : 7262.353467,
"stddev" : 979.853703,
"N" : 766692,
"percentile" : {
"1.000000" : 6816,
"5.000000" : 6944,
"10.000000" : 7008,
"20.000000" : 7072,
"30.000000" : 7072,
"40.000000" : 7136,
"50.000000" : 7136,
"60.000000" : 7200,
"70.000000" : 7264,
"80.000000" : 7264,
"90.000000" : 7328,
"95.000000" : 7456,
"99.000000" : 10432,
"99.500000" : 13632,
"99.900000" : 15680,
"99.950000" : 15808,
"99.990000" : 58112
}
},
"lat_ns" : {
"min" : 6743,
"max" : 79590,
"mean" : 8814.128316,
"stddev" : 984.427864,
"N" : 766692
},
"bw_min" : 54595,
"bw_max" : 55137,
"bw_agg" : 100.000000,
"bw_mean" : 54770.357143,
"bw_dev" : 141.769915,
"bw_samples" : 14,
"iops_min" : 109190,
"iops_max" : 110274,
"iops_mean" : 109540.714286,
"iops_stddev" : 283.539830,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 8.571429,
"sys_cpu" : 34.971429,
"ctx" : 766689,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 98.723738,
"20" : 1.253959,
"50" : 0.010000,
"100" : 0.014217,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 1098692,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 5507,
"write_ticks" : 0,
"in_queue" : 5507,
"util" : 99.069123
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=5437: Wed Oct 26 15:01:31 2022
read: IOPS=110k, BW=53.5MiB/s (56.1MB/s)(374MiB/7001msec)
slat (nsec): min=1452, max=7204, avg=1551.77, stdev=73.37
clat (nsec): min=1042, max=78077, avg=7262.35, stdev=979.85
lat (nsec): min=6743, max=79590, avg=8814.13, stdev=984.43
clat percentiles (nsec):
| 1.00th=[ 6816], 5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7072],
| 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7136], 60.00th=[ 7200],
| 70.00th=[ 7264], 80.00th=[ 7264], 90.00th=[ 7328], 95.00th=[ 7456],
| 99.00th=[10432], 99.50th=[13632], 99.90th=[15680], 99.95th=[15808],
| 99.99th=[58112]
bw ( KiB/s): min=54595, max=55137, per=100.00%, avg=54770.36, stdev=141.77, samples=14
iops : min=109190, max=110274, avg=109540.71, stdev=283.54, samples=14
lat (usec) : 2=0.01%, 4=0.01%, 10=98.72%, 20=1.25%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=8.57%, sys=34.97%, ctx=766689, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=766692,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=53.5MiB/s (56.1MB/s), 53.5MiB/s-53.5MiB/s (56.1MB/s-56.1MB/s), io=374MiB (393MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=1098692/0, merge=0/0, ticks=5507/0, in_queue=5507, util=99.07%
$ cat /tmp/fio-output_512_8_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790726,
"timestamp_ms" : 1666790726420,
"time" : "Wed Oct 26 15:25:26 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1511606784,
"io_kbytes" : 1476178,
"bw_bytes" : 215912981,
"bw" : 210852,
"iops" : 421704.042280,
"runtime" : 7001,
"total_ios" : 2952350,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1333,
"max" : 30808,
"mean" : 1878.538164,
"stddev" : 370.577820,
"N" : 2952350
},
"clat_ns" : {
"min" : 3927,
"max" : 150492,
"mean" : 16780.232220,
"stddev" : 3545.257165,
"N" : 2952357,
"percentile" : {
"1.000000" : 12736,
"5.000000" : 13248,
"10.000000" : 13760,
"20.000000" : 15168,
"30.000000" : 15808,
"40.000000" : 16192,
"50.000000" : 16512,
"60.000000" : 16768,
"70.000000" : 17024,
"80.000000" : 17536,
"90.000000" : 18048,
"95.000000" : 18816,
"99.000000" : 33024,
"99.500000" : 35072,
"99.900000" : 50944,
"99.950000" : 67072,
"99.990000" : 78336
}
},
"lat_ns" : {
"min" : 7194,
"max" : 151965,
"mean" : 18658.769599,
"stddev" : 3523.940859,
"N" : 2952357
},
"bw_min" : 208714,
"bw_max" : 212848,
"bw_agg" : 100.000000,
"bw_mean" : 210911.285714,
"bw_dev" : 1029.007022,
"bw_samples" : 14,
"iops_min" : 417428,
"iops_max" : 425697,
"iops_mean" : 421822.500000,
"iops_stddev" : 2058.121277,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.842857,
"sys_cpu" : 81.442857,
"ctx" : 15090,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999966,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.087591,
"20" : 95.854726,
"50" : 3.952004,
"100" : 0.104730,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 4279306,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 24601,
"write_ticks" : 0,
"in_queue" : 24601,
"util" : 99.069123
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=10583: Wed Oct 26 15:25:26 2022
read: IOPS=422k, BW=206MiB/s (216MB/s)(1442MiB/7001msec)
slat (nsec): min=1333, max=30808, avg=1878.54, stdev=370.58
clat (usec): min=3, max=150, avg=16.78, stdev= 3.55
lat (usec): min=7, max=151, avg=18.66, stdev= 3.52
clat percentiles (nsec):
| 1.00th=[12736], 5.00th=[13248], 10.00th=[13760], 20.00th=[15168],
| 30.00th=[15808], 40.00th=[16192], 50.00th=[16512], 60.00th=[16768],
| 70.00th=[17024], 80.00th=[17536], 90.00th=[18048], 95.00th=[18816],
| 99.00th=[33024], 99.50th=[35072], 99.90th=[50944], 99.95th=[67072],
| 99.99th=[78336]
bw ( KiB/s): min=208714, max=212848, per=100.00%, avg=210911.29, stdev=1029.01, samples=14
iops : min=417428, max=425697, avg=421822.50, stdev=2058.12, samples=14
lat (usec) : 4=0.01%, 10=0.09%, 20=95.85%, 50=3.95%, 100=0.10%
lat (usec) : 250=0.01%
cpu : usr=15.84%, sys=81.44%, ctx=15090, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2952350,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=206MiB/s (216MB/s), 206MiB/s-206MiB/s (216MB/s-216MB/s), io=1442MiB (1512MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=4279306/0, merge=0/0, ticks=24601/0, in_queue=24601, util=99.07%
$ rm /tmp/fio-output_512_8_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790740,
"timestamp_ms" : 1666790740160,
"time" : "Wed Oct 26 15:25:40 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1562410496,
"io_kbytes" : 1525791,
"bw_bytes" : 223169618,
"bw" : 217939,
"iops" : 435877.017569,
"runtime" : 7001,
"total_ios" : 3051575,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1322,
"max" : 37350,
"mean" : 1848.436398,
"stddev" : 360.896377,
"N" : 3051576
},
"clat_ns" : {
"min" : 5079,
"max" : 131407,
"mean" : 16231.104474,
"stddev" : 3574.667520,
"N" : 3051582,
"percentile" : {
"1.000000" : 12480,
"5.000000" : 12992,
"10.000000" : 13248,
"20.000000" : 14784,
"30.000000" : 15296,
"40.000000" : 15424,
"50.000000" : 15808,
"60.000000" : 16064,
"70.000000" : 16512,
"80.000000" : 17280,
"90.000000" : 17792,
"95.000000" : 18560,
"99.000000" : 32640,
"99.500000" : 34560,
"99.900000" : 49408,
"99.950000" : 65280,
"99.990000" : 77312
}
},
"lat_ns" : {
"min" : 6452,
"max" : 132769,
"mean" : 18079.540984,
"stddev" : 3537.875972,
"N" : 3051582
},
"bw_min" : 217144,
"bw_max" : 219141,
"bw_agg" : 100.000000,
"bw_mean" : 217999.642857,
"bw_dev" : 593.977934,
"bw_samples" : 14,
"iops_min" : 434288,
"iops_max" : 438282,
"iops_mean" : 435999.357143,
"iops_stddev" : 1188.013701,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 16.628571,
"sys_cpu" : 80.442857,
"ctx" : 14900,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999967,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.058167,
"20" : 96.003277,
"50" : 3.840115,
"100" : 0.098146,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 4399333,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 25278,
"write_ticks" : 0,
"in_queue" : 25278,
"util" : 99.069123
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=10630: Wed Oct 26 15:25:40 2022
read: IOPS=436k, BW=213MiB/s (223MB/s)(1490MiB/7001msec)
slat (nsec): min=1322, max=37350, avg=1848.44, stdev=360.90
clat (usec): min=5, max=131, avg=16.23, stdev= 3.57
lat (usec): min=6, max=132, avg=18.08, stdev= 3.54
clat percentiles (nsec):
| 1.00th=[12480], 5.00th=[12992], 10.00th=[13248], 20.00th=[14784],
| 30.00th=[15296], 40.00th=[15424], 50.00th=[15808], 60.00th=[16064],
| 70.00th=[16512], 80.00th=[17280], 90.00th=[17792], 95.00th=[18560],
| 99.00th=[32640], 99.50th=[34560], 99.90th=[49408], 99.95th=[65280],
| 99.99th=[77312]
bw ( KiB/s): min=217144, max=219141, per=100.00%, avg=217999.64, stdev=593.98, samples=14
iops : min=434288, max=438282, avg=435999.36, stdev=1188.01, samples=14
lat (usec) : 10=0.06%, 20=96.00%, 50=3.84%, 100=0.10%, 250=0.01%
cpu : usr=16.63%, sys=80.44%, ctx=14900, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3051575,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=213MiB/s (223MB/s), 213MiB/s-213MiB/s (223MB/s-223MB/s), io=1490MiB (1562MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=4399333/0, merge=0/0, ticks=25278/0, in_queue=25278, util=99.07%
$ rm /tmp/fio-output_512_8_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790755,
"timestamp_ms" : 1666790755304,
"time" : "Wed Oct 26 15:25:55 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1789819392,
"io_kbytes" : 1747870,
"bw_bytes" : 255651962,
"bw" : 249660,
"iops" : 499319.240109,
"runtime" : 7001,
"total_ios" : 3495734,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 23364,
"mean" : 152.689076,
"stddev" : 114.264283,
"N" : 3495734
},
"clat_ns" : {
"min" : 6803,
"max" : 131567,
"mean" : 15002.125793,
"stddev" : 4088.111547,
"N" : 3495738,
"percentile" : {
"1.000000" : 9152,
"5.000000" : 12608,
"10.000000" : 12864,
"20.000000" : 13248,
"30.000000" : 13504,
"40.000000" : 13760,
"50.000000" : 14016,
"60.000000" : 14272,
"70.000000" : 14656,
"80.000000" : 15296,
"90.000000" : 18304,
"95.000000" : 21632,
"99.000000" : 33024,
"99.500000" : 34560,
"99.900000" : 51968,
"99.950000" : 67072,
"99.990000" : 78336
}
},
"lat_ns" : {
"min" : 6973,
"max" : 131697,
"mean" : 15154.814849,
"stddev" : 4085.047977,
"N" : 3495738
},
"bw_min" : 247874,
"bw_max" : 252834,
"bw_agg" : 100.000000,
"bw_mean" : 249794.071429,
"bw_dev" : 1473.101410,
"bw_samples" : 14,
"iops_min" : 495748,
"iops_max" : 505668,
"iops_mean" : 499588.285714,
"iops_stddev" : 2946.178762,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 30.042857,
"sys_cpu" : 69.928571,
"ctx" : 8,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.100000,
"4" : 72.377446,
"8" : 27.523175,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 98.713086,
"8" : 1.286914,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 1.834121,
"20" : 91.377834,
"50" : 6.677682,
"100" : 0.109390,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=10686: Wed Oct 26 15:25:55 2022
read: IOPS=499k, BW=244MiB/s (256MB/s)(1707MiB/7001msec)
slat (nsec): min=110, max=23364, avg=152.69, stdev=114.26
clat (usec): min=6, max=131, avg=15.00, stdev= 4.09
lat (usec): min=6, max=131, avg=15.15, stdev= 4.09
clat percentiles (nsec):
| 1.00th=[ 9152], 5.00th=[12608], 10.00th=[12864], 20.00th=[13248],
| 30.00th=[13504], 40.00th=[13760], 50.00th=[14016], 60.00th=[14272],
| 70.00th=[14656], 80.00th=[15296], 90.00th=[18304], 95.00th=[21632],
| 99.00th=[33024], 99.50th=[34560], 99.90th=[51968], 99.95th=[67072],
| 99.99th=[78336]
bw ( KiB/s): min=247874, max=252834, per=100.00%, avg=249794.07, stdev=1473.10, samples=14
iops : min=495748, max=505668, avg=499588.29, stdev=2946.18, samples=14
lat (usec) : 10=1.83%, 20=91.38%, 50=6.68%, 100=0.11%, 250=0.01%
cpu : usr=30.04%, sys=69.93%, ctx=8, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=72.4%, 8=27.5%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=98.7%, 8=1.3%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3495734,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=244MiB/s (256MB/s), 244MiB/s-244MiB/s (256MB/s-256MB/s), io=1707MiB (1790MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790770,
"timestamp_ms" : 1666790770427,
"time" : "Wed Oct 26 15:26:10 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1791554560,
"io_kbytes" : 1749565,
"bw_bytes" : 255899808,
"bw" : 249902,
"iops" : 499803.313812,
"runtime" : 7001,
"total_ios" : 3499123,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 24446,
"mean" : 156.179399,
"stddev" : 115.772836,
"N" : 3499123
},
"clat_ns" : {
"min" : 6753,
"max" : 143268,
"mean" : 14972.242013,
"stddev" : 4066.042433,
"N" : 3499127,
"percentile" : {
"1.000000" : 9152,
"5.000000" : 12480,
"10.000000" : 12864,
"20.000000" : 13248,
"30.000000" : 13504,
"40.000000" : 13760,
"50.000000" : 14016,
"60.000000" : 14272,
"70.000000" : 14656,
"80.000000" : 15168,
"90.000000" : 18304,
"95.000000" : 21632,
"99.000000" : 33024,
"99.500000" : 34560,
"99.900000" : 50944,
"99.950000" : 67072,
"99.990000" : 78336
}
},
"lat_ns" : {
"min" : 6923,
"max" : 143469,
"mean" : 15128.421388,
"stddev" : 4063.050872,
"N" : 3499127
},
"bw_min" : 247329,
"bw_max" : 253134,
"bw_agg" : 100.000000,
"bw_mean" : 250039.500000,
"bw_dev" : 1803.163577,
"bw_samples" : 14,
"iops_min" : 494658,
"iops_max" : 506268,
"iops_mean" : 500079.142857,
"iops_stddev" : 3606.253582,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 29.642857,
"sys_cpu" : 70.328571,
"ctx" : 9,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.117515,
"4" : 72.370934,
"8" : 27.508121,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 98.752327,
"8" : 1.247673,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 1.845948,
"20" : 91.447028,
"50" : 6.605484,
"100" : 0.100740,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=10743: Wed Oct 26 15:26:10 2022
read: IOPS=500k, BW=244MiB/s (256MB/s)(1709MiB/7001msec)
slat (nsec): min=110, max=24446, avg=156.18, stdev=115.77
clat (usec): min=6, max=143, avg=14.97, stdev= 4.07
lat (usec): min=6, max=143, avg=15.13, stdev= 4.06
clat percentiles (nsec):
| 1.00th=[ 9152], 5.00th=[12480], 10.00th=[12864], 20.00th=[13248],
| 30.00th=[13504], 40.00th=[13760], 50.00th=[14016], 60.00th=[14272],
| 70.00th=[14656], 80.00th=[15168], 90.00th=[18304], 95.00th=[21632],
| 99.00th=[33024], 99.50th=[34560], 99.90th=[50944], 99.95th=[67072],
| 99.99th=[78336]
bw ( KiB/s): min=247329, max=253134, per=100.00%, avg=250039.50, stdev=1803.16, samples=14
iops : min=494658, max=506268, avg=500079.14, stdev=3606.25, samples=14
lat (usec) : 10=1.85%, 20=91.45%, 50=6.61%, 100=0.10%, 250=0.01%
cpu : usr=29.64%, sys=70.33%, ctx=9, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=72.4%, 8=27.5%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=98.8%, 8=1.2%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3499123,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=244MiB/s (256MB/s), 244MiB/s-244MiB/s (256MB/s-256MB/s), io=1709MiB (1792MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-reference_2.txt': No such file or directory
$ rm /tmp/fio-output_512_8_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790788,
"timestamp_ms" : 1666790788540,
"time" : "Wed Oct 26 15:26:28 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1763398144,
"io_kbytes" : 1722068,
"bw_bytes" : 251878037,
"bw" : 245974,
"iops" : 491948.435938,
"runtime" : 7001,
"total_ios" : 3444131,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 100,
"max" : 20659,
"mean" : 159.195923,
"stddev" : 134.215657,
"N" : 3444132
},
"clat_ns" : {
"min" : 6412,
"max" : 125806,
"mean" : 15197.289828,
"stddev" : 4131.746276,
"N" : 3444137,
"percentile" : {
"1.000000" : 8896,
"5.000000" : 12608,
"10.000000" : 12992,
"20.000000" : 13376,
"30.000000" : 13632,
"40.000000" : 13888,
"50.000000" : 14144,
"60.000000" : 14528,
"70.000000" : 14912,
"80.000000" : 15552,
"90.000000" : 18816,
"95.000000" : 22400,
"99.000000" : 33024,
"99.500000" : 34560,
"99.900000" : 51456,
"99.950000" : 69120,
"99.990000" : 80384
}
},
"lat_ns" : {
"min" : 6612,
"max" : 126016,
"mean" : 15356.485787,
"stddev" : 4127.680316,
"N" : 3444137
},
"bw_min" : 244609,
"bw_max" : 248021,
"bw_agg" : 100.000000,
"bw_mean" : 246103.357143,
"bw_dev" : 1102.233300,
"bw_samples" : 14,
"iops_min" : 489218,
"iops_max" : 496042,
"iops_mean" : 492206.785714,
"iops_stddev" : 2204.746425,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 32.785714,
"sys_cpu" : 67.185714,
"ctx" : 12,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.129118,
"4" : 72.261014,
"8" : 27.606935,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 98.422303,
"8" : 1.577697,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 1.778126,
"20" : 90.700905,
"50" : 7.415281,
"100" : 0.104874,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=10800: Wed Oct 26 15:26:28 2022
read: IOPS=492k, BW=240MiB/s (252MB/s)(1682MiB/7001msec)
slat (nsec): min=100, max=20659, avg=159.20, stdev=134.22
clat (usec): min=6, max=125, avg=15.20, stdev= 4.13
lat (usec): min=6, max=126, avg=15.36, stdev= 4.13
clat percentiles (nsec):
| 1.00th=[ 8896], 5.00th=[12608], 10.00th=[12992], 20.00th=[13376],
| 30.00th=[13632], 40.00th=[13888], 50.00th=[14144], 60.00th=[14528],
| 70.00th=[14912], 80.00th=[15552], 90.00th=[18816], 95.00th=[22400],
| 99.00th=[33024], 99.50th=[34560], 99.90th=[51456], 99.95th=[69120],
| 99.99th=[80384]
bw ( KiB/s): min=244609, max=248021, per=100.00%, avg=246103.36, stdev=1102.23, samples=14
iops : min=489218, max=496042, avg=492206.79, stdev=2204.75, samples=14
lat (usec) : 10=1.78%, 20=90.70%, 50=7.42%, 100=0.10%, 250=0.01%
cpu : usr=32.79%, sys=67.19%, ctx=12, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=72.3%, 8=27.6%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=98.4%, 8=1.6%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3444131,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=240MiB/s (252MB/s), 240MiB/s-240MiB/s (252MB/s-252MB/s), io=1682MiB (1763MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790805,
"timestamp_ms" : 1666790805897,
"time" : "Wed Oct 26 15:26:45 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1621752320,
"io_kbytes" : 1583742,
"bw_bytes" : 231645810,
"bw" : 226216,
"iops" : 452432.081131,
"runtime" : 7001,
"total_ios" : 3167477,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1563,
"max" : 52558,
"mean" : 1732.091699,
"stddev" : 132.863734,
"N" : 3167478
},
"clat_ns" : {
"min" : 250,
"max" : 126818,
"mean" : 15699.602769,
"stddev" : 2539.253016,
"N" : 3167484,
"percentile" : {
"1.000000" : 14912,
"5.000000" : 14912,
"10.000000" : 15040,
"20.000000" : 15040,
"30.000000" : 15168,
"40.000000" : 15168,
"50.000000" : 15296,
"60.000000" : 15296,
"70.000000" : 15296,
"80.000000" : 15424,
"90.000000" : 15808,
"95.000000" : 16192,
"99.000000" : 28288,
"99.500000" : 30592,
"99.900000" : 43776,
"99.950000" : 60672,
"99.990000" : 74240
}
},
"lat_ns" : {
"min" : 6412,
"max" : 128531,
"mean" : 17431.694401,
"stddev" : 2546.918829,
"N" : 3167484
},
"bw_min" : 224958,
"bw_max" : 227326,
"bw_agg" : 100.000000,
"bw_mean" : 226257.285714,
"bw_dev" : 705.539339,
"bw_samples" : 14,
"iops_min" : 449916,
"iops_max" : 454652,
"iops_mean" : 452514.642857,
"iops_stddev" : 1411.151936,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 30.385714,
"sys_cpu" : 69.600000,
"ctx" : 15,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999984,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.027404,
"20" : 96.891659,
"50" : 3.001000,
"100" : 0.078675,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=10858: Wed Oct 26 15:26:45 2022
read: IOPS=452k, BW=221MiB/s (232MB/s)(1547MiB/7001msec)
slat (nsec): min=1563, max=52558, avg=1732.09, stdev=132.86
clat (nsec): min=250, max=126818, avg=15699.60, stdev=2539.25
lat (usec): min=6, max=128, avg=17.43, stdev= 2.55
clat percentiles (nsec):
| 1.00th=[14912], 5.00th=[14912], 10.00th=[15040], 20.00th=[15040],
| 30.00th=[15168], 40.00th=[15168], 50.00th=[15296], 60.00th=[15296],
| 70.00th=[15296], 80.00th=[15424], 90.00th=[15808], 95.00th=[16192],
| 99.00th=[28288], 99.50th=[30592], 99.90th=[43776], 99.95th=[60672],
| 99.99th=[74240]
bw ( KiB/s): min=224958, max=227326, per=100.00%, avg=226257.29, stdev=705.54, samples=14
iops : min=449916, max=454652, avg=452514.64, stdev=1411.15, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 10=0.03%, 20=96.89%, 50=3.00%, 100=0.08%, 250=0.01%
cpu : usr=30.39%, sys=69.60%, ctx=15, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3167477,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=221MiB/s (232MB/s), 221MiB/s-221MiB/s (232MB/s-232MB/s), io=1547MiB (1622MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790824,
"timestamp_ms" : 1666790824648,
"time" : "Wed Oct 26 15:27:04 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1612016128,
"io_kbytes" : 1574234,
"bw_bytes" : 230255124,
"bw" : 224858,
"iops" : 449715.897729,
"runtime" : 7001,
"total_ios" : 3148461,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1573,
"max" : 40045,
"mean" : 1761.557793,
"stddev" : 123.973748,
"N" : 3148462
},
"clat_ns" : {
"min" : 250,
"max" : 120826,
"mean" : 15794.247972,
"stddev" : 2581.682894,
"N" : 3148468,
"percentile" : {
"1.000000" : 14912,
"5.000000" : 15040,
"10.000000" : 15040,
"20.000000" : 15168,
"30.000000" : 15168,
"40.000000" : 15296,
"50.000000" : 15296,
"60.000000" : 15424,
"70.000000" : 15424,
"80.000000" : 15808,
"90.000000" : 16064,
"95.000000" : 16192,
"99.000000" : 28032,
"99.500000" : 30592,
"99.900000" : 49408,
"99.950000" : 64768,
"99.990000" : 75264
}
},
"lat_ns" : {
"min" : 5831,
"max" : 122500,
"mean" : 17555.805718,
"stddev" : 2589.532167,
"N" : 3148468
},
"bw_min" : 224054,
"bw_max" : 227215,
"bw_agg" : 100.000000,
"bw_mean" : 224976.142857,
"bw_dev" : 840.788902,
"bw_samples" : 14,
"iops_min" : 448108,
"iops_max" : 454430,
"iops_mean" : 449952.428571,
"iops_stddev" : 1681.479647,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 31.585714,
"sys_cpu" : 68.400000,
"ctx" : 11,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999984,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.032619,
"20" : 96.921004,
"50" : 2.948266,
"100" : 0.097254,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=10915: Wed Oct 26 15:27:04 2022
read: IOPS=450k, BW=220MiB/s (230MB/s)(1537MiB/7001msec)
slat (nsec): min=1573, max=40045, avg=1761.56, stdev=123.97
clat (nsec): min=250, max=120826, avg=15794.25, stdev=2581.68
lat (usec): min=5, max=122, avg=17.56, stdev= 2.59
clat percentiles (nsec):
| 1.00th=[14912], 5.00th=[15040], 10.00th=[15040], 20.00th=[15168],
| 30.00th=[15168], 40.00th=[15296], 50.00th=[15296], 60.00th=[15424],
| 70.00th=[15424], 80.00th=[15808], 90.00th=[16064], 95.00th=[16192],
| 99.00th=[28032], 99.50th=[30592], 99.90th=[49408], 99.95th=[64768],
| 99.99th=[75264]
bw ( KiB/s): min=224054, max=227215, per=100.00%, avg=224976.14, stdev=840.79, samples=14
iops : min=448108, max=454430, avg=449952.43, stdev=1681.48, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 10=0.03%, 20=96.92%, 50=2.95%, 100=0.10%, 250=0.01%
cpu : usr=31.59%, sys=68.40%, ctx=11, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3148461,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=220MiB/s (230MB/s), 220MiB/s-220MiB/s (230MB/s-230MB/s), io=1537MiB (1612MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-reference_2.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790841,
"timestamp_ms" : 1666790841375,
"time" : "Wed Oct 26 15:27:21 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1616416768,
"io_kbytes" : 1578532,
"bw_bytes" : 230883697,
"bw" : 225472,
"iops" : 450943.579489,
"runtime" : 7001,
"total_ios" : 3157056,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1572,
"max" : 17793,
"mean" : 1750.871442,
"stddev" : 124.108060,
"N" : 3157057
},
"clat_ns" : {
"min" : 260,
"max" : 128782,
"mean" : 15750.550326,
"stddev" : 2540.179660,
"N" : 3157063,
"percentile" : {
"1.000000" : 14912,
"5.000000" : 15040,
"10.000000" : 15040,
"20.000000" : 15168,
"30.000000" : 15168,
"40.000000" : 15168,
"50.000000" : 15296,
"60.000000" : 15296,
"70.000000" : 15424,
"80.000000" : 15808,
"90.000000" : 15936,
"95.000000" : 16192,
"99.000000" : 28288,
"99.500000" : 30592,
"99.900000" : 44288,
"99.950000" : 63232,
"99.990000" : 74240
}
},
"lat_ns" : {
"min" : 6081,
"max" : 130635,
"mean" : 17501.421643,
"stddev" : 2547.987942,
"N" : 3157063
},
"bw_min" : 224516,
"bw_max" : 226843,
"bw_agg" : 100.000000,
"bw_mean" : 225679.230769,
"bw_dev" : 917.421582,
"bw_samples" : 13,
"iops_min" : 449032,
"iops_max" : 453686,
"iops_mean" : 451358.615385,
"iops_stddev" : 1835.033630,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 31.128571,
"sys_cpu" : 68.857143,
"ctx" : 8,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999984,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.031517,
"20" : 96.933409,
"50" : 2.949868,
"100" : 0.084382,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=10974: Wed Oct 26 15:27:21 2022
read: IOPS=451k, BW=220MiB/s (231MB/s)(1542MiB/7001msec)
slat (nsec): min=1572, max=17793, avg=1750.87, stdev=124.11
clat (nsec): min=260, max=128782, avg=15750.55, stdev=2540.18
lat (usec): min=6, max=130, avg=17.50, stdev= 2.55
clat percentiles (nsec):
| 1.00th=[14912], 5.00th=[15040], 10.00th=[15040], 20.00th=[15168],
| 30.00th=[15168], 40.00th=[15168], 50.00th=[15296], 60.00th=[15296],
| 70.00th=[15424], 80.00th=[15808], 90.00th=[15936], 95.00th=[16192],
| 99.00th=[28288], 99.50th=[30592], 99.90th=[44288], 99.95th=[63232],
| 99.99th=[74240]
bw ( KiB/s): min=224516, max=226843, per=100.00%, avg=225679.23, stdev=917.42, samples=13
iops : min=449032, max=453686, avg=451358.62, stdev=1835.03, samples=13
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.03%, 20=96.93%, 50=2.95%, 100=0.08%
lat (usec) : 250=0.01%
cpu : usr=31.13%, sys=68.86%, ctx=8, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3157056,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=220MiB/s (231MB/s), 220MiB/s-220MiB/s (231MB/s-231MB/s), io=1542MiB (1616MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790856,
"timestamp_ms" : 1666790856790,
"time" : "Wed Oct 26 15:27:36 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1419834368,
"io_kbytes" : 1386557,
"bw_bytes" : 202804509,
"bw" : 198051,
"iops" : 396101.414084,
"runtime" : 7001,
"total_ios" : 2773106,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1393,
"max" : 70382,
"mean" : 2032.974013,
"stddev" : 315.234496,
"N" : 2773107
},
"clat_ns" : {
"min" : 4167,
"max" : 134191,
"mean" : 17867.042998,
"stddev" : 3010.820945,
"N" : 2773113,
"percentile" : {
"1.000000" : 13376,
"5.000000" : 15424,
"10.000000" : 16768,
"20.000000" : 17024,
"30.000000" : 17280,
"40.000000" : 17536,
"50.000000" : 17536,
"60.000000" : 17536,
"70.000000" : 18048,
"80.000000" : 18048,
"90.000000" : 18304,
"95.000000" : 19584,
"99.000000" : 31616,
"99.500000" : 34560,
"99.900000" : 50432,
"99.950000" : 64256,
"99.990000" : 77312
}
},
"lat_ns" : {
"min" : 6372,
"max" : 135744,
"mean" : 19900.015943,
"stddev" : 2965.963506,
"N" : 2773113
},
"bw_min" : 197423,
"bw_max" : 198917,
"bw_agg" : 100.000000,
"bw_mean" : 198149.500000,
"bw_dev" : 471.310897,
"bw_samples" : 14,
"iops_min" : 394847,
"iops_max" : 397834,
"iops_mean" : 396299.142857,
"iops_stddev" : 942.487532,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 20.971429,
"sys_cpu" : 76.928571,
"ctx" : 16494,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999982,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.102989,
"20" : 95.387482,
"50" : 4.409135,
"100" : 0.099780,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=11022: Wed Oct 26 15:27:36 2022
read: IOPS=396k, BW=193MiB/s (203MB/s)(1354MiB/7001msec)
slat (nsec): min=1393, max=70382, avg=2032.97, stdev=315.23
clat (usec): min=4, max=134, avg=17.87, stdev= 3.01
lat (usec): min=6, max=135, avg=19.90, stdev= 2.97
clat percentiles (nsec):
| 1.00th=[13376], 5.00th=[15424], 10.00th=[16768], 20.00th=[17024],
| 30.00th=[17280], 40.00th=[17536], 50.00th=[17536], 60.00th=[17536],
| 70.00th=[18048], 80.00th=[18048], 90.00th=[18304], 95.00th=[19584],
| 99.00th=[31616], 99.50th=[34560], 99.90th=[50432], 99.95th=[64256],
| 99.99th=[77312]
bw ( KiB/s): min=197423, max=198917, per=100.00%, avg=198149.50, stdev=471.31, samples=14
iops : min=394847, max=397834, avg=396299.14, stdev=942.49, samples=14
lat (usec) : 10=0.10%, 20=95.39%, 50=4.41%, 100=0.10%, 250=0.01%
cpu : usr=20.97%, sys=76.93%, ctx=16494, majf=0, minf=1
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2773106,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=193MiB/s (203MB/s), 193MiB/s-193MiB/s (203MB/s-203MB/s), io=1354MiB (1420MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790873,
"timestamp_ms" : 1666790873848,
"time" : "Wed Oct 26 15:27:53 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1414161408,
"io_kbytes" : 1381017,
"bw_bytes" : 201994201,
"bw" : 197259,
"iops" : 394518.925868,
"runtime" : 7001,
"total_ios" : 2762027,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1392,
"max" : 30357,
"mean" : 2037.367052,
"stddev" : 294.648162,
"N" : 2762027
},
"clat_ns" : {
"min" : 4098,
"max" : 135283,
"mean" : 17936.435806,
"stddev" : 3001.136611,
"N" : 2762034,
"percentile" : {
"1.000000" : 13376,
"5.000000" : 15552,
"10.000000" : 16768,
"20.000000" : 17280,
"30.000000" : 17536,
"40.000000" : 17536,
"50.000000" : 17536,
"60.000000" : 17536,
"70.000000" : 18048,
"80.000000" : 18304,
"90.000000" : 18304,
"95.000000" : 19584,
"99.000000" : 31616,
"99.500000" : 35072,
"99.900000" : 51456,
"99.950000" : 66048,
"99.990000" : 76288
}
},
"lat_ns" : {
"min" : 7033,
"max" : 136786,
"mean" : 19973.801927,
"stddev" : 2955.591710,
"N" : 2762034
},
"bw_min" : 196032,
"bw_max" : 198923,
"bw_agg" : 100.000000,
"bw_mean" : 197370.428571,
"bw_dev" : 916.608608,
"bw_samples" : 14,
"iops_min" : 392064,
"iops_max" : 397846,
"iops_mean" : 394741.000000,
"iops_stddev" : 1833.052687,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 20.842857,
"sys_cpu" : 77.085714,
"ctx" : 16725,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999982,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.081208,
"20" : 95.414129,
"50" : 4.391956,
"100" : 0.112671,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=11070: Wed Oct 26 15:27:53 2022
read: IOPS=395k, BW=193MiB/s (202MB/s)(1349MiB/7001msec)
slat (nsec): min=1392, max=30357, avg=2037.37, stdev=294.65
clat (usec): min=4, max=135, avg=17.94, stdev= 3.00
lat (usec): min=7, max=136, avg=19.97, stdev= 2.96
clat percentiles (nsec):
| 1.00th=[13376], 5.00th=[15552], 10.00th=[16768], 20.00th=[17280],
| 30.00th=[17536], 40.00th=[17536], 50.00th=[17536], 60.00th=[17536],
| 70.00th=[18048], 80.00th=[18304], 90.00th=[18304], 95.00th=[19584],
| 99.00th=[31616], 99.50th=[35072], 99.90th=[51456], 99.95th=[66048],
| 99.99th=[76288]
bw ( KiB/s): min=196032, max=198923, per=100.00%, avg=197370.43, stdev=916.61, samples=14
iops : min=392064, max=397846, avg=394741.00, stdev=1833.05, samples=14
lat (usec) : 10=0.08%, 20=95.41%, 50=4.39%, 100=0.11%, 250=0.01%
cpu : usr=20.84%, sys=77.09%, ctx=16725, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2762027,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=193MiB/s (202MB/s), 193MiB/s-193MiB/s (202MB/s-202MB/s), io=1349MiB (1414MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789304,
"timestamp_ms" : 1666789304886,
"time" : "Wed Oct 26 15:01:44 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 393098240,
"io_kbytes" : 383885,
"bw_bytes" : 56148870,
"bw" : 54832,
"iops" : 109665.762034,
"runtime" : 7001,
"total_ios" : 767770,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1493,
"max" : 6332,
"mean" : 1588.558791,
"stddev" : 74.080772,
"N" : 767770
},
"clat_ns" : {
"min" : 2945,
"max" : 114725,
"mean" : 7224.469301,
"stddev" : 999.991459,
"N" : 767770,
"percentile" : {
"1.000000" : 6752,
"5.000000" : 6944,
"10.000000" : 6944,
"20.000000" : 7008,
"30.000000" : 7008,
"40.000000" : 7072,
"50.000000" : 7136,
"60.000000" : 7136,
"70.000000" : 7200,
"80.000000" : 7200,
"90.000000" : 7328,
"95.000000" : 7456,
"99.000000" : 10432,
"99.500000" : 13632,
"99.900000" : 15552,
"99.950000" : 15808,
"99.990000" : 59136
}
},
"lat_ns" : {
"min" : 7434,
"max" : 116268,
"mean" : 8813.028092,
"stddev" : 1004.849320,
"N" : 767770
},
"bw_min" : 54684,
"bw_max" : 55007,
"bw_agg" : 100.000000,
"bw_mean" : 54870.642857,
"bw_dev" : 100.171090,
"bw_samples" : 14,
"iops_min" : 109368,
"iops_max" : 110014,
"iops_mean" : 109741.500000,
"iops_stddev" : 200.451701,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 7.971429,
"sys_cpu" : 35.742857,
"ctx" : 767772,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 98.707556,
"20" : 1.272517,
"50" : 0.010000,
"100" : 0.014588,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 1100470,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 5516,
"write_ticks" : 0,
"in_queue" : 5516,
"util" : 99.108381
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=5483: Wed Oct 26 15:01:44 2022
read: IOPS=110k, BW=53.5MiB/s (56.1MB/s)(375MiB/7001msec)
slat (nsec): min=1493, max=6332, avg=1588.56, stdev=74.08
clat (nsec): min=2945, max=114725, avg=7224.47, stdev=999.99
lat (usec): min=7, max=116, avg= 8.81, stdev= 1.00
clat percentiles (nsec):
| 1.00th=[ 6752], 5.00th=[ 6944], 10.00th=[ 6944], 20.00th=[ 7008],
| 30.00th=[ 7008], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7136],
| 70.00th=[ 7200], 80.00th=[ 7200], 90.00th=[ 7328], 95.00th=[ 7456],
| 99.00th=[10432], 99.50th=[13632], 99.90th=[15552], 99.95th=[15808],
| 99.99th=[59136]
bw ( KiB/s): min=54684, max=55007, per=100.00%, avg=54870.64, stdev=100.17, samples=14
iops : min=109368, max=110014, avg=109741.50, stdev=200.45, samples=14
lat (usec) : 4=0.01%, 10=98.71%, 20=1.27%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%
cpu : usr=7.97%, sys=35.74%, ctx=767772, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=767770,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=53.5MiB/s (56.1MB/s), 53.5MiB/s-53.5MiB/s (56.1MB/s-56.1MB/s), io=375MiB (393MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=1100470/0, merge=0/0, ticks=5516/0, in_queue=5516, util=99.11%
$ cat /tmp/fio-output_512_8_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790891,
"timestamp_ms" : 1666790891701,
"time" : "Wed Oct 26 15:28:11 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1422672384,
"io_kbytes" : 1389328,
"bw_bytes" : 203209882,
"bw" : 198447,
"iops" : 396893.158120,
"runtime" : 7001,
"total_ios" : 2778649,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1402,
"max" : 29996,
"mean" : 2024.455626,
"stddev" : 305.377303,
"N" : 2778650
},
"clat_ns" : {
"min" : 2254,
"max" : 131687,
"mean" : 17831.837688,
"stddev" : 3036.634295,
"N" : 2778656,
"percentile" : {
"1.000000" : 13248,
"5.000000" : 15168,
"10.000000" : 16512,
"20.000000" : 17024,
"30.000000" : 17280,
"40.000000" : 17536,
"50.000000" : 17536,
"60.000000" : 17536,
"70.000000" : 17536,
"80.000000" : 18048,
"90.000000" : 18304,
"95.000000" : 19584,
"99.000000" : 31872,
"99.500000" : 35072,
"99.900000" : 50944,
"99.950000" : 64768,
"99.990000" : 77312
}
},
"lat_ns" : {
"min" : 6422,
"max" : 133160,
"mean" : 19856.292749,
"stddev" : 2987.817199,
"N" : 2778656
},
"bw_min" : 197185,
"bw_max" : 199877,
"bw_agg" : 100.000000,
"bw_mean" : 198482.571429,
"bw_dev" : 752.010404,
"bw_samples" : 14,
"iops_min" : 394370,
"iops_max" : 399754,
"iops_mean" : 396965.142857,
"iops_stddev" : 1504.020809,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 22.071429,
"sys_cpu" : 75.800000,
"ctx" : 16640,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999982,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.090620,
"20" : 95.460888,
"50" : 4.343226,
"100" : 0.104331,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=11118: Wed Oct 26 15:28:11 2022
read: IOPS=397k, BW=194MiB/s (203MB/s)(1357MiB/7001msec)
slat (nsec): min=1402, max=29996, avg=2024.46, stdev=305.38
clat (usec): min=2, max=131, avg=17.83, stdev= 3.04
lat (usec): min=6, max=133, avg=19.86, stdev= 2.99
clat percentiles (nsec):
| 1.00th=[13248], 5.00th=[15168], 10.00th=[16512], 20.00th=[17024],
| 30.00th=[17280], 40.00th=[17536], 50.00th=[17536], 60.00th=[17536],
| 70.00th=[17536], 80.00th=[18048], 90.00th=[18304], 95.00th=[19584],
| 99.00th=[31872], 99.50th=[35072], 99.90th=[50944], 99.95th=[64768],
| 99.99th=[77312]
bw ( KiB/s): min=197185, max=199877, per=100.00%, avg=198482.57, stdev=752.01, samples=14
iops : min=394370, max=399754, avg=396965.14, stdev=1504.02, samples=14
lat (usec) : 4=0.01%, 10=0.09%, 20=95.46%, 50=4.34%, 100=0.10%
lat (usec) : 250=0.01%
cpu : usr=22.07%, sys=75.80%, ctx=16640, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2778649,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=194MiB/s (203MB/s), 194MiB/s-194MiB/s (203MB/s-203MB/s), io=1357MiB (1423MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790907,
"timestamp_ms" : 1666790907820,
"time" : "Wed Oct 26 15:28:27 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1573419008,
"io_kbytes" : 1536542,
"bw_bytes" : 224742037,
"bw" : 219474,
"iops" : 438948.293101,
"runtime" : 7001,
"total_ios" : 3073077,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1442,
"max" : 20178,
"mean" : 1602.113093,
"stddev" : 122.876606,
"N" : 3073077
},
"clat_ns" : {
"min" : 4669,
"max" : 139171,
"mean" : 16380.314330,
"stddev" : 3042.824860,
"N" : 3073084,
"percentile" : {
"1.000000" : 13888,
"5.000000" : 15552,
"10.000000" : 15680,
"20.000000" : 15680,
"30.000000" : 15680,
"40.000000" : 15808,
"50.000000" : 15808,
"60.000000" : 15808,
"70.000000" : 15936,
"80.000000" : 16320,
"90.000000" : 16512,
"95.000000" : 17536,
"99.000000" : 31616,
"99.500000" : 35584,
"99.900000" : 51456,
"99.950000" : 66048,
"99.990000" : 78336
}
},
"lat_ns" : {
"min" : 6172,
"max" : 140724,
"mean" : 17982.427350,
"stddev" : 3044.762520,
"N" : 3073084
},
"bw_min" : 218423,
"bw_max" : 220928,
"bw_agg" : 100.000000,
"bw_mean" : 219536.928571,
"bw_dev" : 658.497411,
"bw_samples" : 14,
"iops_min" : 436846,
"iops_max" : 441857,
"iops_mean" : 439073.928571,
"iops_stddev" : 1317.157339,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 30.700000,
"sys_cpu" : 67.314286,
"ctx" : 18155,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999967,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.426283,
"20" : 95.925061,
"50" : 3.544721,
"100" : 0.103024,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 4442260,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 25258,
"write_ticks" : 0,
"in_queue" : 25258,
"util" : 99.069123
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=11167: Wed Oct 26 15:28:27 2022
read: IOPS=439k, BW=214MiB/s (225MB/s)(1501MiB/7001msec)
slat (nsec): min=1442, max=20178, avg=1602.11, stdev=122.88
clat (usec): min=4, max=139, avg=16.38, stdev= 3.04
lat (usec): min=6, max=140, avg=17.98, stdev= 3.04
clat percentiles (nsec):
| 1.00th=[13888], 5.00th=[15552], 10.00th=[15680], 20.00th=[15680],
| 30.00th=[15680], 40.00th=[15808], 50.00th=[15808], 60.00th=[15808],
| 70.00th=[15936], 80.00th=[16320], 90.00th=[16512], 95.00th=[17536],
| 99.00th=[31616], 99.50th=[35584], 99.90th=[51456], 99.95th=[66048],
| 99.99th=[78336]
bw ( KiB/s): min=218423, max=220928, per=100.00%, avg=219536.93, stdev=658.50, samples=14
iops : min=436846, max=441857, avg=439073.93, stdev=1317.16, samples=14
lat (usec) : 10=0.43%, 20=95.93%, 50=3.54%, 100=0.10%, 250=0.01%
cpu : usr=30.70%, sys=67.31%, ctx=18155, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3073077,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=214MiB/s (225MB/s), 214MiB/s-214MiB/s (225MB/s-225MB/s), io=1501MiB (1573MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=4442260/0, merge=0/0, ticks=25258/0, in_queue=25258, util=99.07%
$ rm /tmp/fio-output_512_8_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790924,
"timestamp_ms" : 1666790924175,
"time" : "Wed Oct 26 15:28:44 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1567867904,
"io_kbytes" : 1531121,
"bw_bytes" : 223949136,
"bw" : 218700,
"iops" : 437399.514355,
"runtime" : 7001,
"total_ios" : 3062234,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1442,
"max" : 80601,
"mean" : 1606.787758,
"stddev" : 104.510126,
"N" : 3062235
},
"clat_ns" : {
"min" : 4960,
"max" : 135585,
"mean" : 16440.666159,
"stddev" : 2924.034150,
"N" : 3062241,
"percentile" : {
"1.000000" : 13760,
"5.000000" : 15680,
"10.000000" : 15680,
"20.000000" : 15808,
"30.000000" : 15808,
"40.000000" : 15808,
"50.000000" : 15936,
"60.000000" : 15936,
"70.000000" : 16064,
"80.000000" : 16512,
"90.000000" : 16512,
"95.000000" : 17536,
"99.000000" : 31104,
"99.500000" : 35072,
"99.900000" : 48896,
"99.950000" : 63744,
"99.990000" : 78336
}
},
"lat_ns" : {
"min" : 6483,
"max" : 137268,
"mean" : 18047.453991,
"stddev" : 2925.765710,
"N" : 3062241
},
"bw_min" : 218165,
"bw_max" : 219778,
"bw_agg" : 100.000000,
"bw_mean" : 218823.571429,
"bw_dev" : 414.416868,
"bw_samples" : 14,
"iops_min" : 436330,
"iops_max" : 439557,
"iops_mean" : 437647.285714,
"iops_stddev" : 829.026624,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 31.600000,
"sys_cpu" : 66.528571,
"ctx" : 17763,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999967,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.421882,
"20" : 96.022054,
"50" : 3.461231,
"100" : 0.094180,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 4419891,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 25004,
"write_ticks" : 0,
"in_queue" : 25004,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=11213: Wed Oct 26 15:28:44 2022
read: IOPS=437k, BW=214MiB/s (224MB/s)(1495MiB/7001msec)
slat (nsec): min=1442, max=80601, avg=1606.79, stdev=104.51
clat (usec): min=4, max=135, avg=16.44, stdev= 2.92
lat (usec): min=6, max=137, avg=18.05, stdev= 2.93
clat percentiles (nsec):
| 1.00th=[13760], 5.00th=[15680], 10.00th=[15680], 20.00th=[15808],
| 30.00th=[15808], 40.00th=[15808], 50.00th=[15936], 60.00th=[15936],
| 70.00th=[16064], 80.00th=[16512], 90.00th=[16512], 95.00th=[17536],
| 99.00th=[31104], 99.50th=[35072], 99.90th=[48896], 99.95th=[63744],
| 99.99th=[78336]
bw ( KiB/s): min=218165, max=219778, per=100.00%, avg=218823.57, stdev=414.42, samples=14
iops : min=436330, max=439557, avg=437647.29, stdev=829.03, samples=14
lat (usec) : 10=0.42%, 20=96.02%, 50=3.46%, 100=0.09%, 250=0.01%
cpu : usr=31.60%, sys=66.53%, ctx=17763, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3062234,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=214MiB/s (224MB/s), 214MiB/s-214MiB/s (224MB/s-224MB/s), io=1495MiB (1568MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=4419891/0, merge=0/0, ticks=25004/0, in_queue=25004, util=99.09%
$ rm /tmp/fio-output_512_8_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790941,
"timestamp_ms" : 1666790941607,
"time" : "Wed Oct 26 15:29:01 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1575713792,
"io_kbytes" : 1538783,
"bw_bytes" : 225069817,
"bw" : 219794,
"iops" : 439588.344522,
"runtime" : 7001,
"total_ios" : 3077558,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1442,
"max" : 20608,
"mean" : 1603.968562,
"stddev" : 112.982422,
"N" : 3077559
},
"clat_ns" : {
"min" : 4729,
"max" : 131707,
"mean" : 16357.507495,
"stddev" : 3033.941500,
"N" : 3077565,
"percentile" : {
"1.000000" : 13632,
"5.000000" : 15552,
"10.000000" : 15552,
"20.000000" : 15680,
"30.000000" : 15680,
"40.000000" : 15808,
"50.000000" : 15808,
"60.000000" : 15808,
"70.000000" : 15936,
"80.000000" : 16320,
"90.000000" : 16512,
"95.000000" : 17536,
"99.000000" : 31872,
"99.500000" : 36096,
"99.900000" : 49408,
"99.950000" : 64256,
"99.990000" : 77312
}
},
"lat_ns" : {
"min" : 6242,
"max" : 133490,
"mean" : 17961.475945,
"stddev" : 3035.114182,
"N" : 3077565
},
"bw_min" : 219076,
"bw_max" : 222226,
"bw_agg" : 100.000000,
"bw_mean" : 219857.071429,
"bw_dev" : 825.530735,
"bw_samples" : 14,
"iops_min" : 438152,
"iops_max" : 444452,
"iops_mean" : 439714.285714,
"iops_stddev" : 1651.104219,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 30.700000,
"sys_cpu" : 67.271429,
"ctx" : 18039,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.440284,
"20" : 95.900321,
"50" : 3.560713,
"100" : 0.098650,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 4443543,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 25340,
"write_ticks" : 0,
"in_queue" : 25340,
"util" : 99.069123
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=11259: Wed Oct 26 15:29:01 2022
read: IOPS=440k, BW=215MiB/s (225MB/s)(1503MiB/7001msec)
slat (nsec): min=1442, max=20608, avg=1603.97, stdev=112.98
clat (usec): min=4, max=131, avg=16.36, stdev= 3.03
lat (usec): min=6, max=133, avg=17.96, stdev= 3.04
clat percentiles (nsec):
| 1.00th=[13632], 5.00th=[15552], 10.00th=[15552], 20.00th=[15680],
| 30.00th=[15680], 40.00th=[15808], 50.00th=[15808], 60.00th=[15808],
| 70.00th=[15936], 80.00th=[16320], 90.00th=[16512], 95.00th=[17536],
| 99.00th=[31872], 99.50th=[36096], 99.90th=[49408], 99.95th=[64256],
| 99.99th=[77312]
bw ( KiB/s): min=219076, max=222226, per=100.00%, avg=219857.07, stdev=825.53, samples=14
iops : min=438152, max=444452, avg=439714.29, stdev=1651.10, samples=14
lat (usec) : 10=0.44%, 20=95.90%, 50=3.56%, 100=0.10%, 250=0.01%
cpu : usr=30.70%, sys=67.27%, ctx=18039, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3077558,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=215MiB/s (225MB/s), 215MiB/s-215MiB/s (225MB/s-225MB/s), io=1503MiB (1576MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=4443543/0, merge=0/0, ticks=25340/0, in_queue=25340, util=99.07%
$ rm /tmp/fio-output_512_1_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt': No such file or directory
$ rm /tmp/fio-output_512_8_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-bdev_aio_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790960,
"timestamp_ms" : 1666790960594,
"time" : "Wed Oct 26 15:29:20 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1661046784,
"io_kbytes" : 1622116,
"bw_bytes" : 237258503,
"bw" : 231697,
"iops" : 463394.372233,
"runtime" : 7001,
"total_ios" : 3244224,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1633,
"max" : 22713,
"mean" : 1818.265835,
"stddev" : 123.789028,
"N" : 3244225
},
"clat_ns" : {
"min" : 361,
"max" : 109205,
"mean" : 10578.592328,
"stddev" : 4535.960378,
"N" : 3244224,
"percentile" : {
"1.000000" : 4192,
"5.000000" : 4320,
"10.000000" : 6112,
"20.000000" : 6304,
"30.000000" : 8160,
"40.000000" : 8384,
"50.000000" : 10432,
"60.000000" : 11200,
"70.000000" : 12864,
"80.000000" : 14528,
"90.000000" : 14784,
"95.000000" : 15296,
"99.000000" : 26496,
"99.500000" : 28288,
"99.900000" : 43776,
"99.950000" : 60160,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 5740,
"max" : 111099,
"mean" : 12396.858125,
"stddev" : 4531.767359,
"N" : 3244224
},
"bw_min" : 230450,
"bw_max" : 233465,
"bw_agg" : 100.000000,
"bw_mean" : 231740.142857,
"bw_dev" : 820.889841,
"bw_samples" : 14,
"iops_min" : 460900,
"iops_max" : 466930,
"iops_mean" : 463480.285714,
"iops_stddev" : 1641.459736,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 36.971429,
"sys_cpu" : 63.014286,
"ctx" : 5,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 8.133347,
"4" : 72.486425,
"8" : 19.379550,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 85.202722,
"8" : 14.797278,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 43.187308,
"20" : 54.000433,
"50" : 2.732826,
"100" : 0.077307,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=11316: Wed Oct 26 15:29:20 2022
read: IOPS=463k, BW=226MiB/s (237MB/s)(1584MiB/7001msec)
slat (nsec): min=1633, max=22713, avg=1818.27, stdev=123.79
clat (nsec): min=361, max=109205, avg=10578.59, stdev=4535.96
lat (usec): min=5, max=111, avg=12.40, stdev= 4.53
clat percentiles (nsec):
| 1.00th=[ 4192], 5.00th=[ 4320], 10.00th=[ 6112], 20.00th=[ 6304],
| 30.00th=[ 8160], 40.00th=[ 8384], 50.00th=[10432], 60.00th=[11200],
| 70.00th=[12864], 80.00th=[14528], 90.00th=[14784], 95.00th=[15296],
| 99.00th=[26496], 99.50th=[28288], 99.90th=[43776], 99.95th=[60160],
| 99.99th=[72192]
bw ( KiB/s): min=230450, max=233465, per=100.00%, avg=231740.14, stdev=820.89, samples=14
iops : min=460900, max=466930, avg=463480.29, stdev=1641.46, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=43.19%, 20=54.00%, 50=2.73%, 100=0.08%
lat (usec) : 250=0.01%
cpu : usr=36.97%, sys=63.01%, ctx=5, majf=0, minf=0
IO depths : 1=0.1%, 2=8.1%, 4=72.5%, 8=19.4%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=85.2%, 8=14.8%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3244224,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=226MiB/s (237MB/s), 226MiB/s-226MiB/s (237MB/s-237MB/s), io=1584MiB (1661MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-bdev_aio_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790977,
"timestamp_ms" : 1666790977538,
"time" : "Wed Oct 26 15:29:37 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1666486272,
"io_kbytes" : 1627428,
"bw_bytes" : 238035462,
"bw" : 232456,
"iops" : 464912.012570,
"runtime" : 7001,
"total_ios" : 3254849,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1623,
"max" : 23605,
"mean" : 1803.010349,
"stddev" : 126.458209,
"N" : 3254850
},
"clat_ns" : {
"min" : 370,
"max" : 125115,
"mean" : 10935.840299,
"stddev" : 4372.129142,
"N" : 3254856,
"percentile" : {
"1.000000" : 4256,
"5.000000" : 6112,
"10.000000" : 6176,
"20.000000" : 6560,
"30.000000" : 8160,
"40.000000" : 10048,
"50.000000" : 10688,
"60.000000" : 12480,
"70.000000" : 12736,
"80.000000" : 14400,
"90.000000" : 14656,
"95.000000" : 15168,
"99.000000" : 27008,
"99.500000" : 28800,
"99.900000" : 43264,
"99.950000" : 60672,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 5661,
"max" : 126928,
"mean" : 12738.850612,
"stddev" : 4363.905228,
"N" : 3254856
},
"bw_min" : 229362,
"bw_max" : 234011,
"bw_agg" : 100.000000,
"bw_mean" : 232498.785714,
"bw_dev" : 1354.748752,
"bw_samples" : 14,
"iops_min" : 458724,
"iops_max" : 468022,
"iops_mean" : 464997.642857,
"iops_stddev" : 2709.526147,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 41.585714,
"sys_cpu" : 58.385714,
"ctx" : 7,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 1.864664,
"4" : 77.562246,
"8" : 20.571922,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 84.411387,
"8" : 15.588613,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 39.686818,
"20" : 57.371356,
"50" : 2.863297,
"100" : 0.075702,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=11373: Wed Oct 26 15:29:37 2022
read: IOPS=465k, BW=227MiB/s (238MB/s)(1589MiB/7001msec)
slat (nsec): min=1623, max=23605, avg=1803.01, stdev=126.46
clat (nsec): min=370, max=125115, avg=10935.84, stdev=4372.13
lat (usec): min=5, max=126, avg=12.74, stdev= 4.36
clat percentiles (nsec):
| 1.00th=[ 4256], 5.00th=[ 6112], 10.00th=[ 6176], 20.00th=[ 6560],
| 30.00th=[ 8160], 40.00th=[10048], 50.00th=[10688], 60.00th=[12480],
| 70.00th=[12736], 80.00th=[14400], 90.00th=[14656], 95.00th=[15168],
| 99.00th=[27008], 99.50th=[28800], 99.90th=[43264], 99.95th=[60672],
| 99.99th=[72192]
bw ( KiB/s): min=229362, max=234011, per=100.00%, avg=232498.79, stdev=1354.75, samples=14
iops : min=458724, max=468022, avg=464997.64, stdev=2709.53, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=39.69%, 20=57.37%, 50=2.86%, 100=0.08%
lat (usec) : 250=0.01%
cpu : usr=41.59%, sys=58.39%, ctx=7, majf=0, minf=0
IO depths : 1=0.1%, 2=1.9%, 4=77.6%, 8=20.6%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=84.4%, 8=15.6%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3254849,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=1589MiB (1666MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-bdev_aio_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666790992,
"timestamp_ms" : 1666790992810,
"time" : "Wed Oct 26 15:29:52 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1667806208,
"io_kbytes" : 1628717,
"bw_bytes" : 238223997,
"bw" : 232640,
"iops" : 465280.531353,
"runtime" : 7001,
"total_ios" : 3257429,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1623,
"max" : 23795,
"mean" : 1807.492342,
"stddev" : 121.710490,
"N" : 3257430
},
"clat_ns" : {
"min" : 391,
"max" : 110808,
"mean" : 10528.263130,
"stddev" : 4533.572349,
"N" : 3257434,
"percentile" : {
"1.000000" : 4192,
"5.000000" : 4256,
"10.000000" : 6112,
"20.000000" : 6240,
"30.000000" : 8096,
"40.000000" : 8256,
"50.000000" : 10304,
"60.000000" : 11200,
"70.000000" : 12736,
"80.000000" : 14528,
"90.000000" : 14784,
"95.000000" : 15296,
"99.000000" : 26240,
"99.500000" : 28288,
"99.900000" : 43264,
"99.950000" : 60672,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 5650,
"max" : 112531,
"mean" : 12335.755434,
"stddev" : 4529.016440,
"N" : 3257434
},
"bw_min" : 230862,
"bw_max" : 235114,
"bw_agg" : 100.000000,
"bw_mean" : 232693.000000,
"bw_dev" : 1547.197367,
"bw_samples" : 14,
"iops_min" : 461716,
"iops_max" : 470228,
"iops_mean" : 465386.000000,
"iops_stddev" : 3095.091101,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 38.057143,
"sys_cpu" : 61.928571,
"ctx" : 4,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 8.325922,
"4" : 72.295359,
"8" : 19.377736,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 85.258211,
"8" : 14.741789,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 43.861094,
"20" : 53.338047,
"50" : 2.724081,
"100" : 0.074261,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=11431: Wed Oct 26 15:29:52 2022
read: IOPS=465k, BW=227MiB/s (238MB/s)(1591MiB/7001msec)
slat (nsec): min=1623, max=23795, avg=1807.49, stdev=121.71
clat (nsec): min=391, max=110808, avg=10528.26, stdev=4533.57
lat (usec): min=5, max=112, avg=12.34, stdev= 4.53
clat percentiles (nsec):
| 1.00th=[ 4192], 5.00th=[ 4256], 10.00th=[ 6112], 20.00th=[ 6240],
| 30.00th=[ 8096], 40.00th=[ 8256], 50.00th=[10304], 60.00th=[11200],
| 70.00th=[12736], 80.00th=[14528], 90.00th=[14784], 95.00th=[15296],
| 99.00th=[26240], 99.50th=[28288], 99.90th=[43264], 99.95th=[60672],
| 99.99th=[72192]
bw ( KiB/s): min=230862, max=235114, per=100.00%, avg=232693.00, stdev=1547.20, samples=14
iops : min=461716, max=470228, avg=465386.00, stdev=3095.09, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=43.86%, 20=53.34%, 50=2.72%, 100=0.07%
lat (usec) : 250=0.01%
cpu : usr=38.06%, sys=61.93%, ctx=4, majf=0, minf=0
IO depths : 1=0.1%, 2=8.3%, 4=72.3%, 8=19.4%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=85.3%, 8=14.7%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3257429,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=1591MiB (1668MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666791008,
"timestamp_ms" : 1666791008196,
"time" : "Wed Oct 26 15:30:08 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1567925248,
"io_kbytes" : 1531177,
"bw_bytes" : 223957327,
"bw" : 218708,
"iops" : 437415.940580,
"runtime" : 7001,
"total_ios" : 3062349,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1663,
"max" : 34314,
"mean" : 1858.695535,
"stddev" : 122.078812,
"N" : 3062350
},
"clat_ns" : {
"min" : 831,
"max" : 122730,
"mean" : 11448.851285,
"stddev" : 4591.776289,
"N" : 3062354,
"percentile" : {
"1.000000" : 4704,
"5.000000" : 4896,
"10.000000" : 6688,
"20.000000" : 6880,
"30.000000" : 8768,
"40.000000" : 9152,
"50.000000" : 11584,
"60.000000" : 12352,
"70.000000" : 13888,
"80.000000" : 15680,
"90.000000" : 16064,
"95.000000" : 16768,
"99.000000" : 26240,
"99.500000" : 29056,
"99.900000" : 44800,
"99.950000" : 59648,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 5661,
"max" : 124534,
"mean" : 13307.546791,
"stddev" : 4584.897019,
"N" : 3062354
},
"bw_min" : 218159,
"bw_max" : 219551,
"bw_agg" : 100.000000,
"bw_mean" : 218814.142857,
"bw_dev" : 384.929535,
"bw_samples" : 14,
"iops_min" : 436318,
"iops_max" : 439102,
"iops_mean" : 437628.428571,
"iops_stddev" : 769.682623,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 39.185714,
"sys_cpu" : 60.785714,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 7.306744,
"4" : 73.195772,
"8" : 19.496700,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 85.097695,
"8" : 14.902305,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 42.621955,
"20" : 54.869220,
"50" : 2.423107,
"100" : 0.082584,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=11488: Wed Oct 26 15:30:08 2022
read: IOPS=437k, BW=214MiB/s (224MB/s)(1495MiB/7001msec)
slat (nsec): min=1663, max=34314, avg=1858.70, stdev=122.08
clat (nsec): min=831, max=122730, avg=11448.85, stdev=4591.78
lat (usec): min=5, max=124, avg=13.31, stdev= 4.58
clat percentiles (nsec):
| 1.00th=[ 4704], 5.00th=[ 4896], 10.00th=[ 6688], 20.00th=[ 6880],
| 30.00th=[ 8768], 40.00th=[ 9152], 50.00th=[11584], 60.00th=[12352],
| 70.00th=[13888], 80.00th=[15680], 90.00th=[16064], 95.00th=[16768],
| 99.00th=[26240], 99.50th=[29056], 99.90th=[44800], 99.95th=[59648],
| 99.99th=[72192]
bw ( KiB/s): min=218159, max=219551, per=100.00%, avg=218814.14, stdev=384.93, samples=14
iops : min=436318, max=439102, avg=437628.43, stdev=769.68, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 4=0.01%, 10=42.62%, 20=54.87%, 50=2.42%, 100=0.08%
lat (usec) : 250=0.01%
cpu : usr=39.19%, sys=60.79%, ctx=10, majf=0, minf=0
IO depths : 1=0.1%, 2=7.3%, 4=73.2%, 8=19.5%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=85.1%, 8=14.9%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3062349,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=214MiB/s (224MB/s), 214MiB/s-214MiB/s (224MB/s-224MB/s), io=1495MiB (1568MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666791023,
"timestamp_ms" : 1666791023513,
"time" : "Wed Oct 26 15:30:23 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1576878592,
"io_kbytes" : 1539920,
"bw_bytes" : 225236193,
"bw" : 219957,
"iops" : 439913.726610,
"runtime" : 7001,
"total_ios" : 3079836,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1663,
"max" : 19526,
"mean" : 1853.081294,
"stddev" : 134.461061,
"N" : 3079837
},
"clat_ns" : {
"min" : 812,
"max" : 111990,
"mean" : 11416.888518,
"stddev" : 4588.974480,
"N" : 3079841,
"percentile" : {
"1.000000" : 4704,
"5.000000" : 4896,
"10.000000" : 6624,
"20.000000" : 6816,
"30.000000" : 8768,
"40.000000" : 9152,
"50.000000" : 11584,
"60.000000" : 12352,
"70.000000" : 13888,
"80.000000" : 15552,
"90.000000" : 15936,
"95.000000" : 16512,
"99.000000" : 26752,
"99.500000" : 29312,
"99.900000" : 44288,
"99.950000" : 60160,
"99.990000" : 73216
}
},
"lat_ns" : {
"min" : 5660,
"max" : 113774,
"mean" : 13269.969731,
"stddev" : 4580.597825,
"N" : 3079841
},
"bw_min" : 219083,
"bw_max" : 221465,
"bw_agg" : 100.000000,
"bw_mean" : 219996.928571,
"bw_dev" : 649.316737,
"bw_samples" : 14,
"iops_min" : 438166,
"iops_max" : 442931,
"iops_mean" : 439993.928571,
"iops_stddev" : 1298.760028,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 37.185714,
"sys_cpu" : 62.800000,
"ctx" : 5,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 6.747048,
"4" : 73.657980,
"8" : 19.593803,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 85.015234,
"8" : 14.984766,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 42.298421,
"20" : 55.114980,
"50" : 2.504776,
"100" : 0.078511,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=11545: Wed Oct 26 15:30:23 2022
read: IOPS=440k, BW=215MiB/s (225MB/s)(1504MiB/7001msec)
slat (nsec): min=1663, max=19526, avg=1853.08, stdev=134.46
clat (nsec): min=812, max=111990, avg=11416.89, stdev=4588.97
lat (usec): min=5, max=113, avg=13.27, stdev= 4.58
clat percentiles (nsec):
| 1.00th=[ 4704], 5.00th=[ 4896], 10.00th=[ 6624], 20.00th=[ 6816],
| 30.00th=[ 8768], 40.00th=[ 9152], 50.00th=[11584], 60.00th=[12352],
| 70.00th=[13888], 80.00th=[15552], 90.00th=[15936], 95.00th=[16512],
| 99.00th=[26752], 99.50th=[29312], 99.90th=[44288], 99.95th=[60160],
| 99.99th=[73216]
bw ( KiB/s): min=219083, max=221465, per=100.00%, avg=219996.93, stdev=649.32, samples=14
iops : min=438166, max=442931, avg=439993.93, stdev=1298.76, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 4=0.01%, 10=42.30%, 20=55.11%, 50=2.50%, 100=0.08%
lat (usec) : 250=0.01%
cpu : usr=37.19%, sys=62.80%, ctx=5, majf=0, minf=0
IO depths : 1=0.1%, 2=6.7%, 4=73.7%, 8=19.6%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=85.0%, 8=15.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3079836,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=215MiB/s (225MB/s), 215MiB/s-215MiB/s (225MB/s-225MB/s), io=1504MiB (1577MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666791039,
"timestamp_ms" : 1666791039010,
"time" : "Wed Oct 26 15:30:39 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1597813760,
"io_kbytes" : 1560365,
"bw_bytes" : 228226504,
"bw" : 222877,
"iops" : 445754.320811,
"runtime" : 7001,
"total_ios" : 3120726,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1653,
"max" : 42329,
"mean" : 1834.530434,
"stddev" : 111.510378,
"N" : 3120727
},
"clat_ns" : {
"min" : 791,
"max" : 125785,
"mean" : 11024.351078,
"stddev" : 4765.287623,
"N" : 3120730,
"percentile" : {
"1.000000" : 4640,
"5.000000" : 4704,
"10.000000" : 4896,
"20.000000" : 6688,
"30.000000" : 8640,
"40.000000" : 8768,
"50.000000" : 10688,
"60.000000" : 11712,
"70.000000" : 13632,
"80.000000" : 15424,
"90.000000" : 15808,
"95.000000" : 16320,
"99.000000" : 27520,
"99.500000" : 29824,
"99.900000" : 45312,
"99.950000" : 61184,
"99.990000" : 74240
}
},
"lat_ns" : {
"min" : 5721,
"max" : 127559,
"mean" : 12858.881482,
"stddev" : 4762.175625,
"N" : 3120730
},
"bw_min" : 221033,
"bw_max" : 224499,
"bw_agg" : 100.000000,
"bw_mean" : 222992.500000,
"bw_dev" : 1021.068049,
"bw_samples" : 14,
"iops_min" : 442066,
"iops_max" : 448998,
"iops_mean" : 445985.000000,
"iops_stddev" : 2042.136099,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 36.557143,
"sys_cpu" : 63.414286,
"ctx" : 13,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 10.780793,
"4" : 70.423421,
"8" : 18.795178,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 85.618620,
"8" : 14.381380,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 44.750004,
"20" : 52.543575,
"50" : 2.621249,
"100" : 0.082577,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=11603: Wed Oct 26 15:30:39 2022
read: IOPS=446k, BW=218MiB/s (228MB/s)(1524MiB/7001msec)
slat (nsec): min=1653, max=42329, avg=1834.53, stdev=111.51
clat (nsec): min=791, max=125785, avg=11024.35, stdev=4765.29
lat (usec): min=5, max=127, avg=12.86, stdev= 4.76
clat percentiles (nsec):
| 1.00th=[ 4640], 5.00th=[ 4704], 10.00th=[ 4896], 20.00th=[ 6688],
| 30.00th=[ 8640], 40.00th=[ 8768], 50.00th=[10688], 60.00th=[11712],
| 70.00th=[13632], 80.00th=[15424], 90.00th=[15808], 95.00th=[16320],
| 99.00th=[27520], 99.50th=[29824], 99.90th=[45312], 99.95th=[61184],
| 99.99th=[74240]
bw ( KiB/s): min=221033, max=224499, per=100.00%, avg=222992.50, stdev=1021.07, samples=14
iops : min=442066, max=448998, avg=445985.00, stdev=2042.14, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 4=0.01%, 10=44.75%, 20=52.54%, 50=2.62%, 100=0.08%
lat (usec) : 250=0.01%
cpu : usr=36.56%, sys=63.41%, ctx=13, majf=0, minf=0
IO depths : 1=0.1%, 2=10.8%, 4=70.4%, 8=18.8%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=85.6%, 8=14.4%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3120726,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=218MiB/s (228MB/s), 218MiB/s-218MiB/s (228MB/s-228MB/s), io=1524MiB (1598MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789319,
"timestamp_ms" : 1666789319922,
"time" : "Wed Oct 26 15:01:59 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 484563456,
"io_kbytes" : 473206,
"bw_bytes" : 69213463,
"bw" : 67591,
"iops" : 135182.545351,
"runtime" : 7001,
"total_ios" : 946413,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 150,
"max" : 9447,
"mean" : 172.883611,
"stddev" : 23.288240,
"N" : 946413
},
"clat_ns" : {
"min" : 6261,
"max" : 85470,
"mean" : 6992.056989,
"stddev" : 1186.274005,
"N" : 946413,
"percentile" : {
"1.000000" : 6496,
"5.000000" : 6688,
"10.000000" : 6752,
"20.000000" : 6752,
"30.000000" : 6752,
"40.000000" : 6816,
"50.000000" : 6816,
"60.000000" : 6880,
"70.000000" : 6880,
"80.000000" : 6944,
"90.000000" : 7008,
"95.000000" : 7136,
"99.000000" : 13120,
"99.500000" : 14784,
"99.900000" : 17024,
"99.950000" : 17280,
"99.990000" : 60672
}
},
"lat_ns" : {
"min" : 6432,
"max" : 85640,
"mean" : 7164.940600,
"stddev" : 1186.749310,
"N" : 946413
},
"bw_min" : 67403,
"bw_max" : 67839,
"bw_agg" : 100.000000,
"bw_mean" : 67627.785714,
"bw_dev" : 111.226850,
"bw_samples" : 14,
"iops_min" : 134806,
"iops_max" : 135679,
"iops_mean" : 135255.642857,
"iops_stddev" : 222.599885,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 53.014286,
"sys_cpu" : 46.957143,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 98.381151,
"20" : 1.599090,
"50" : 0.010000,
"100" : 0.015955,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=5540: Wed Oct 26 15:01:59 2022
read: IOPS=135k, BW=66.0MiB/s (69.2MB/s)(462MiB/7001msec)
slat (nsec): min=150, max=9447, avg=172.88, stdev=23.29
clat (nsec): min=6261, max=85470, avg=6992.06, stdev=1186.27
lat (nsec): min=6432, max=85640, avg=7164.94, stdev=1186.75
clat percentiles (nsec):
| 1.00th=[ 6496], 5.00th=[ 6688], 10.00th=[ 6752], 20.00th=[ 6752],
| 30.00th=[ 6752], 40.00th=[ 6816], 50.00th=[ 6816], 60.00th=[ 6880],
| 70.00th=[ 6880], 80.00th=[ 6944], 90.00th=[ 7008], 95.00th=[ 7136],
| 99.00th=[13120], 99.50th=[14784], 99.90th=[17024], 99.95th=[17280],
| 99.99th=[60672]
bw ( KiB/s): min=67403, max=67839, per=100.00%, avg=67627.79, stdev=111.23, samples=14
iops : min=134806, max=135679, avg=135255.64, stdev=222.60, samples=14
lat (usec) : 10=98.38%, 20=1.60%, 50=0.01%, 100=0.02%
cpu : usr=53.01%, sys=46.96%, ctx=10, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=946413,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=66.0MiB/s (69.2MB/s), 66.0MiB/s-66.0MiB/s (69.2MB/s-69.2MB/s), io=462MiB (485MB), run=7001-7001msec
$ cat /tmp/fio-output_512_8_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666791053,
"timestamp_ms" : 1666791053029,
"time" : "Wed Oct 26 15:30:53 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1495874560,
"io_kbytes" : 1460815,
"bw_bytes" : 213665842,
"bw" : 208658,
"iops" : 417315.097843,
"runtime" : 7001,
"total_ios" : 2921623,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1512,
"max" : 36950,
"mean" : 1657.647678,
"stddev" : 106.702955,
"N" : 2921623
},
"clat_ns" : {
"min" : 5039,
"max" : 134552,
"mean" : 17269.205602,
"stddev" : 2767.694361,
"N" : 2921630,
"percentile" : {
"1.000000" : 14912,
"5.000000" : 16512,
"10.000000" : 16512,
"20.000000" : 16768,
"30.000000" : 16768,
"40.000000" : 16768,
"50.000000" : 16768,
"60.000000" : 16768,
"70.000000" : 16768,
"80.000000" : 17280,
"90.000000" : 17536,
"95.000000" : 18048,
"99.000000" : 30336,
"99.500000" : 34560,
"99.900000" : 50944,
"99.950000" : 67072,
"99.990000" : 80384
}
},
"lat_ns" : {
"min" : 6592,
"max" : 136265,
"mean" : 18926.853252,
"stddev" : 2770.487130,
"N" : 2921630
},
"bw_min" : 208125,
"bw_max" : 209317,
"bw_agg" : 100.000000,
"bw_mean" : 208769.428571,
"bw_dev" : 418.942389,
"bw_samples" : 14,
"iops_min" : 416250,
"iops_max" : 418634,
"iops_mean" : 417538.857143,
"iops_stddev" : 837.884778,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 34.928571,
"sys_cpu" : 63.642857,
"ctx" : 16304,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999983,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.213888,
"20" : 96.208785,
"50" : 3.475089,
"100" : 0.101450,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=11651: Wed Oct 26 15:30:53 2022
read: IOPS=417k, BW=204MiB/s (214MB/s)(1427MiB/7001msec)
slat (nsec): min=1512, max=36950, avg=1657.65, stdev=106.70
clat (usec): min=5, max=134, avg=17.27, stdev= 2.77
lat (usec): min=6, max=136, avg=18.93, stdev= 2.77
clat percentiles (nsec):
| 1.00th=[14912], 5.00th=[16512], 10.00th=[16512], 20.00th=[16768],
| 30.00th=[16768], 40.00th=[16768], 50.00th=[16768], 60.00th=[16768],
| 70.00th=[16768], 80.00th=[17280], 90.00th=[17536], 95.00th=[18048],
| 99.00th=[30336], 99.50th=[34560], 99.90th=[50944], 99.95th=[67072],
| 99.99th=[80384]
bw ( KiB/s): min=208125, max=209317, per=100.00%, avg=208769.43, stdev=418.94, samples=14
iops : min=416250, max=418634, avg=417538.86, stdev=837.88, samples=14
lat (usec) : 10=0.21%, 20=96.21%, 50=3.48%, 100=0.10%, 250=0.01%
cpu : usr=34.93%, sys=63.64%, ctx=16304, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2921623,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=204MiB/s (214MB/s), 204MiB/s-204MiB/s (214MB/s-214MB/s), io=1427MiB (1496MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666791066,
"timestamp_ms" : 1666791066967,
"time" : "Wed Oct 26 15:31:06 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1492745216,
"io_kbytes" : 1457759,
"bw_bytes" : 213218856,
"bw" : 208221,
"iops" : 416442.079703,
"runtime" : 7001,
"total_ios" : 2915511,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1502,
"max" : 33263,
"mean" : 1671.443877,
"stddev" : 126.111968,
"N" : 2915511
},
"clat_ns" : {
"min" : 5090,
"max" : 134011,
"mean" : 17299.230179,
"stddev" : 2704.916413,
"N" : 2915518,
"percentile" : {
"1.000000" : 15040,
"5.000000" : 16512,
"10.000000" : 16512,
"20.000000" : 16768,
"30.000000" : 16768,
"40.000000" : 16768,
"50.000000" : 16768,
"60.000000" : 16768,
"70.000000" : 17024,
"80.000000" : 17280,
"90.000000" : 17536,
"95.000000" : 18048,
"99.000000" : 30336,
"99.500000" : 34560,
"99.900000" : 47360,
"99.950000" : 64256,
"99.990000" : 79360
}
},
"lat_ns" : {
"min" : 6653,
"max" : 135624,
"mean" : 18970.673988,
"stddev" : 2707.653440,
"N" : 2915518
},
"bw_min" : 207578,
"bw_max" : 209884,
"bw_agg" : 100.000000,
"bw_mean" : 208259.785714,
"bw_dev" : 724.737107,
"bw_samples" : 14,
"iops_min" : 415156,
"iops_max" : 419769,
"iops_mean" : 416519.500000,
"iops_stddev" : 1449.767660,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 30.614286,
"sys_cpu" : 67.985714,
"ctx" : 16266,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999983,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.206173,
"20" : 96.180395,
"50" : 3.523842,
"100" : 0.088767,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=11700: Wed Oct 26 15:31:06 2022
read: IOPS=416k, BW=203MiB/s (213MB/s)(1424MiB/7001msec)
slat (nsec): min=1502, max=33263, avg=1671.44, stdev=126.11
clat (usec): min=5, max=134, avg=17.30, stdev= 2.70
lat (usec): min=6, max=135, avg=18.97, stdev= 2.71
clat percentiles (nsec):
| 1.00th=[15040], 5.00th=[16512], 10.00th=[16512], 20.00th=[16768],
| 30.00th=[16768], 40.00th=[16768], 50.00th=[16768], 60.00th=[16768],
| 70.00th=[17024], 80.00th=[17280], 90.00th=[17536], 95.00th=[18048],
| 99.00th=[30336], 99.50th=[34560], 99.90th=[47360], 99.95th=[64256],
| 99.99th=[79360]
bw ( KiB/s): min=207578, max=209884, per=100.00%, avg=208259.79, stdev=724.74, samples=14
iops : min=415156, max=419769, avg=416519.50, stdev=1449.77, samples=14
lat (usec) : 10=0.21%, 20=96.18%, 50=3.52%, 100=0.09%, 250=0.01%
cpu : usr=30.61%, sys=67.99%, ctx=16266, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2915511,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=203MiB/s (213MB/s), 203MiB/s-203MiB/s (213MB/s-213MB/s), io=1424MiB (1493MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666791080,
"timestamp_ms" : 1666791080852,
"time" : "Wed Oct 26 15:31:20 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1497201664,
"io_kbytes" : 1462111,
"bw_bytes" : 213855401,
"bw" : 208843,
"iops" : 417685.187830,
"runtime" : 7001,
"total_ios" : 2924214,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1513,
"max" : 25247,
"mean" : 1654.304958,
"stddev" : 100.956893,
"N" : 2924215
},
"clat_ns" : {
"min" : 5080,
"max" : 137948,
"mean" : 17253.342018,
"stddev" : 2675.005836,
"N" : 2924221,
"percentile" : {
"1.000000" : 14784,
"5.000000" : 16768,
"10.000000" : 16768,
"20.000000" : 16768,
"30.000000" : 16768,
"40.000000" : 16768,
"50.000000" : 16768,
"60.000000" : 16768,
"70.000000" : 17024,
"80.000000" : 17024,
"90.000000" : 17536,
"95.000000" : 17792,
"99.000000" : 29568,
"99.500000" : 33536,
"99.900000" : 50944,
"99.950000" : 66048,
"99.990000" : 79360
}
},
"lat_ns" : {
"min" : 6703,
"max" : 139541,
"mean" : 18907.646970,
"stddev" : 2674.174544,
"N" : 2924221
},
"bw_min" : 208370,
"bw_max" : 209183,
"bw_agg" : 100.000000,
"bw_mean" : 208880.357143,
"bw_dev" : 273.334849,
"bw_samples" : 14,
"iops_min" : 416741,
"iops_max" : 418366,
"iops_mean" : 417760.785714,
"iops_stddev" : 546.526117,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 32.342857,
"sys_cpu" : 66.300000,
"ctx" : 15925,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999983,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.234285,
"20" : 96.261628,
"50" : 3.402384,
"100" : 0.100574,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=11749: Wed Oct 26 15:31:20 2022
read: IOPS=418k, BW=204MiB/s (214MB/s)(1428MiB/7001msec)
slat (nsec): min=1513, max=25247, avg=1654.30, stdev=100.96
clat (usec): min=5, max=137, avg=17.25, stdev= 2.68
lat (usec): min=6, max=139, avg=18.91, stdev= 2.67
clat percentiles (nsec):
| 1.00th=[14784], 5.00th=[16768], 10.00th=[16768], 20.00th=[16768],
| 30.00th=[16768], 40.00th=[16768], 50.00th=[16768], 60.00th=[16768],
| 70.00th=[17024], 80.00th=[17024], 90.00th=[17536], 95.00th=[17792],
| 99.00th=[29568], 99.50th=[33536], 99.90th=[50944], 99.95th=[66048],
| 99.99th=[79360]
bw ( KiB/s): min=208370, max=209183, per=100.00%, avg=208880.36, stdev=273.33, samples=14
iops : min=416741, max=418366, avg=417760.79, stdev=546.53, samples=14
lat (usec) : 10=0.23%, 20=96.26%, 50=3.40%, 100=0.10%, 250=0.01%
cpu : usr=32.34%, sys=66.30%, ctx=15925, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2924214,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=204MiB/s (214MB/s), 204MiB/s-204MiB/s (214MB/s-214MB/s), io=1428MiB (1497MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789335,
"timestamp_ms" : 1666789335250,
"time" : "Wed Oct 26 15:02:15 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 483598848,
"io_kbytes" : 472264,
"bw_bytes" : 69075681,
"bw" : 67456,
"iops" : 134913.440937,
"runtime" : 7001,
"total_ios" : 944529,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 160,
"max" : 10830,
"mean" : 176.276544,
"stddev" : 42.021704,
"N" : 944529
},
"clat_ns" : {
"min" : 6261,
"max" : 115436,
"mean" : 6989.813789,
"stddev" : 1151.700203,
"N" : 944529,
"percentile" : {
"1.000000" : 6496,
"5.000000" : 6688,
"10.000000" : 6752,
"20.000000" : 6752,
"30.000000" : 6752,
"40.000000" : 6816,
"50.000000" : 6816,
"60.000000" : 6880,
"70.000000" : 6880,
"80.000000" : 6944,
"90.000000" : 7008,
"95.000000" : 7136,
"99.000000" : 13120,
"99.500000" : 14656,
"99.900000" : 17024,
"99.950000" : 17280,
"99.990000" : 58624
}
},
"lat_ns" : {
"min" : 6432,
"max" : 115607,
"mean" : 7166.090333,
"stddev" : 1152.795263,
"N" : 944529
},
"bw_min" : 67337,
"bw_max" : 67822,
"bw_agg" : 100.000000,
"bw_mean" : 67490.142857,
"bw_dev" : 143.619614,
"bw_samples" : 14,
"iops_min" : 134674,
"iops_max" : 135645,
"iops_mean" : 134980.500000,
"iops_stddev" : 287.344272,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 54.442857,
"sys_cpu" : 45.528571,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 98.367546,
"20" : 1.614985,
"50" : 0.010000,
"100" : 0.013128,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=5597: Wed Oct 26 15:02:15 2022
read: IOPS=135k, BW=65.9MiB/s (69.1MB/s)(461MiB/7001msec)
slat (nsec): min=160, max=10830, avg=176.28, stdev=42.02
clat (usec): min=6, max=115, avg= 6.99, stdev= 1.15
lat (usec): min=6, max=115, avg= 7.17, stdev= 1.15
clat percentiles (nsec):
| 1.00th=[ 6496], 5.00th=[ 6688], 10.00th=[ 6752], 20.00th=[ 6752],
| 30.00th=[ 6752], 40.00th=[ 6816], 50.00th=[ 6816], 60.00th=[ 6880],
| 70.00th=[ 6880], 80.00th=[ 6944], 90.00th=[ 7008], 95.00th=[ 7136],
| 99.00th=[13120], 99.50th=[14656], 99.90th=[17024], 99.95th=[17280],
| 99.99th=[58624]
bw ( KiB/s): min=67337, max=67822, per=100.00%, avg=67490.14, stdev=143.62, samples=14
iops : min=134674, max=135645, avg=134980.50, stdev=287.34, samples=14
lat (usec) : 10=98.37%, 20=1.61%, 50=0.01%, 100=0.01%, 250=0.01%
cpu : usr=54.44%, sys=45.53%, ctx=10, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=944529,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=65.9MiB/s (69.1MB/s), 65.9MiB/s-65.9MiB/s (69.1MB/s-69.1MB/s), io=461MiB (484MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789350,
"timestamp_ms" : 1666789350610,
"time" : "Wed Oct 26 15:02:30 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 482136064,
"io_kbytes" : 470836,
"bw_bytes" : 68866742,
"bw" : 67252,
"iops" : 134505.356378,
"runtime" : 7001,
"total_ios" : 941672,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 160,
"max" : 9859,
"mean" : 177.184497,
"stddev" : 34.333593,
"N" : 941672
},
"clat_ns" : {
"min" : 6301,
"max" : 114083,
"mean" : 7024.269096,
"stddev" : 1168.593145,
"N" : 941672,
"percentile" : {
"1.000000" : 6560,
"5.000000" : 6752,
"10.000000" : 6752,
"20.000000" : 6816,
"30.000000" : 6816,
"40.000000" : 6816,
"50.000000" : 6880,
"60.000000" : 6880,
"70.000000" : 6944,
"80.000000" : 6944,
"90.000000" : 7072,
"95.000000" : 7136,
"99.000000" : 13120,
"99.500000" : 14656,
"99.900000" : 17024,
"99.950000" : 17280,
"99.990000" : 59136
}
},
"lat_ns" : {
"min" : 6472,
"max" : 114264,
"mean" : 7201.453593,
"stddev" : 1169.346311,
"N" : 941672
},
"bw_min" : 67168,
"bw_max" : 67432,
"bw_agg" : 100.000000,
"bw_mean" : 67286.928571,
"bw_dev" : 82.405624,
"bw_samples" : 14,
"iops_min" : 134336,
"iops_max" : 134865,
"iops_mean" : 134574.000000,
"iops_stddev" : 164.977854,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 58.614286,
"sys_cpu" : 41.357143,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 98.349638,
"20" : 1.631885,
"50" : 0.010000,
"100" : 0.014124,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=5654: Wed Oct 26 15:02:30 2022
read: IOPS=135k, BW=65.7MiB/s (68.9MB/s)(460MiB/7001msec)
slat (nsec): min=160, max=9859, avg=177.18, stdev=34.33
clat (usec): min=6, max=114, avg= 7.02, stdev= 1.17
lat (usec): min=6, max=114, avg= 7.20, stdev= 1.17
clat percentiles (nsec):
| 1.00th=[ 6560], 5.00th=[ 6752], 10.00th=[ 6752], 20.00th=[ 6816],
| 30.00th=[ 6816], 40.00th=[ 6816], 50.00th=[ 6880], 60.00th=[ 6880],
| 70.00th=[ 6944], 80.00th=[ 6944], 90.00th=[ 7072], 95.00th=[ 7136],
| 99.00th=[13120], 99.50th=[14656], 99.90th=[17024], 99.95th=[17280],
| 99.99th=[59136]
bw ( KiB/s): min=67168, max=67432, per=100.00%, avg=67286.93, stdev=82.41, samples=14
iops : min=134336, max=134865, avg=134574.00, stdev=164.98, samples=14
lat (usec) : 10=98.35%, 20=1.63%, 50=0.01%, 100=0.01%, 250=0.01%
cpu : usr=58.61%, sys=41.36%, ctx=10, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=941672,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=65.7MiB/s (68.9MB/s), 65.7MiB/s-65.7MiB/s (68.9MB/s-68.9MB/s), io=460MiB (482MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789366,
"timestamp_ms" : 1666789366545,
"time" : "Wed Oct 26 15:02:46 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 551760384,
"io_kbytes" : 538828,
"bw_bytes" : 78811653,
"bw" : 76964,
"iops" : 153928.867305,
"runtime" : 7001,
"total_ios" : 1077656,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1673,
"max" : 19667,
"mean" : 1817.265533,
"stddev" : 149.648202,
"N" : 1077657
},
"clat_ns" : {
"min" : 240,
"max" : 99747,
"mean" : 4449.052741,
"stddev" : 1090.885033,
"N" : 1077656,
"percentile" : {
"1.000000" : 4016,
"5.000000" : 4192,
"10.000000" : 4256,
"20.000000" : 4256,
"30.000000" : 4256,
"40.000000" : 4320,
"50.000000" : 4320,
"60.000000" : 4320,
"70.000000" : 4384,
"80.000000" : 4384,
"90.000000" : 4448,
"95.000000" : 4512,
"99.000000" : 9664,
"99.500000" : 10688,
"99.900000" : 15296,
"99.950000" : 15424,
"99.990000" : 57088
}
},
"lat_ns" : {
"min" : 5521,
"max" : 101600,
"mean" : 6266.318333,
"stddev" : 1102.612816,
"N" : 1077656
},
"bw_min" : 76849,
"bw_max" : 77209,
"bw_agg" : 100.000000,
"bw_mean" : 77006.785714,
"bw_dev" : 107.468907,
"bw_samples" : 14,
"iops_min" : 153699,
"iops_max" : 154418,
"iops_mean" : 154013.714286,
"iops_stddev" : 214.803104,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 16.271429,
"sys_cpu" : 83.714286,
"ctx" : 7,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.958284,
"10" : 98.165184,
"20" : 0.857880,
"50" : 0.010000,
"100" : 0.012249,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=5712: Wed Oct 26 15:02:46 2022
read: IOPS=154k, BW=75.2MiB/s (78.8MB/s)(526MiB/7001msec)
slat (nsec): min=1673, max=19667, avg=1817.27, stdev=149.65
clat (nsec): min=240, max=99747, avg=4449.05, stdev=1090.89
lat (usec): min=5, max=101, avg= 6.27, stdev= 1.10
clat percentiles (nsec):
| 1.00th=[ 4016], 5.00th=[ 4192], 10.00th=[ 4256], 20.00th=[ 4256],
| 30.00th=[ 4256], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4320],
| 70.00th=[ 4384], 80.00th=[ 4384], 90.00th=[ 4448], 95.00th=[ 4512],
| 99.00th=[ 9664], 99.50th=[10688], 99.90th=[15296], 99.95th=[15424],
| 99.99th=[57088]
bw ( KiB/s): min=76849, max=77209, per=100.00%, avg=77006.79, stdev=107.47, samples=14
iops : min=153699, max=154418, avg=154013.71, stdev=214.80, samples=14
lat (nsec) : 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 2=0.01%, 4=0.96%, 10=98.17%, 20=0.86%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=16.27%, sys=83.71%, ctx=7, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1077656,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=75.2MiB/s (78.8MB/s), 75.2MiB/s-75.2MiB/s (78.8MB/s-78.8MB/s), io=526MiB (552MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789382,
"timestamp_ms" : 1666789382237,
"time" : "Wed Oct 26 15:03:02 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 552491520,
"io_kbytes" : 539542,
"bw_bytes" : 78916086,
"bw" : 77066,
"iops" : 154132.981003,
"runtime" : 7001,
"total_ios" : 1079085,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1643,
"max" : 21780,
"mean" : 1806.536104,
"stddev" : 149.212612,
"N" : 1079085
},
"clat_ns" : {
"min" : 260,
"max" : 101110,
"mean" : 4453.978682,
"stddev" : 1094.494781,
"N" : 1079085,
"percentile" : {
"1.000000" : 4016,
"5.000000" : 4192,
"10.000000" : 4256,
"20.000000" : 4256,
"30.000000" : 4256,
"40.000000" : 4320,
"50.000000" : 4320,
"60.000000" : 4320,
"70.000000" : 4384,
"80.000000" : 4384,
"90.000000" : 4448,
"95.000000" : 4576,
"99.000000" : 9664,
"99.500000" : 10688,
"99.900000" : 15296,
"99.950000" : 15552,
"99.990000" : 57088
}
},
"lat_ns" : {
"min" : 5510,
"max" : 102933,
"mean" : 6260.514786,
"stddev" : 1106.115216,
"N" : 1079085
},
"bw_min" : 76941,
"bw_max" : 77411,
"bw_agg" : 100.000000,
"bw_mean" : 77108.928571,
"bw_dev" : 145.888506,
"bw_samples" : 14,
"iops_min" : 153883,
"iops_max" : 154822,
"iops_mean" : 154218.000000,
"iops_stddev" : 291.839524,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.014286,
"sys_cpu" : 82.971429,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.922634,
"10" : 98.195230,
"20" : 0.862768,
"50" : 0.010000,
"100" : 0.012140,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=5770: Wed Oct 26 15:03:02 2022
read: IOPS=154k, BW=75.3MiB/s (78.9MB/s)(527MiB/7001msec)
slat (nsec): min=1643, max=21780, avg=1806.54, stdev=149.21
clat (nsec): min=260, max=101110, avg=4453.98, stdev=1094.49
lat (usec): min=5, max=102, avg= 6.26, stdev= 1.11
clat percentiles (nsec):
| 1.00th=[ 4016], 5.00th=[ 4192], 10.00th=[ 4256], 20.00th=[ 4256],
| 30.00th=[ 4256], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4320],
| 70.00th=[ 4384], 80.00th=[ 4384], 90.00th=[ 4448], 95.00th=[ 4576],
| 99.00th=[ 9664], 99.50th=[10688], 99.90th=[15296], 99.95th=[15552],
| 99.99th=[57088]
bw ( KiB/s): min=76941, max=77411, per=100.00%, avg=77108.93, stdev=145.89, samples=14
iops : min=153883, max=154822, avg=154218.00, stdev=291.84, samples=14
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.92%, 10=98.20%, 20=0.86%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=17.01%, sys=82.97%, ctx=10, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1079085,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=75.3MiB/s (78.9MB/s), 75.3MiB/s-75.3MiB/s (78.9MB/s-78.9MB/s), io=527MiB (552MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789397,
"timestamp_ms" : 1666789397412,
"time" : "Wed Oct 26 15:03:17 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 549262848,
"io_kbytes" : 536389,
"bw_bytes" : 78454913,
"bw" : 76616,
"iops" : 153232.252535,
"runtime" : 7001,
"total_ios" : 1072779,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1663,
"max" : 17694,
"mean" : 1826.517569,
"stddev" : 127.585417,
"N" : 1072779
},
"clat_ns" : {
"min" : 250,
"max" : 114004,
"mean" : 4474.261006,
"stddev" : 1097.446190,
"N" : 1072779,
"percentile" : {
"1.000000" : 4016,
"5.000000" : 4256,
"10.000000" : 4256,
"20.000000" : 4256,
"30.000000" : 4320,
"40.000000" : 4320,
"50.000000" : 4320,
"60.000000" : 4384,
"70.000000" : 4384,
"80.000000" : 4448,
"90.000000" : 4448,
"95.000000" : 4576,
"99.000000" : 9792,
"99.500000" : 10688,
"99.900000" : 15296,
"99.950000" : 15424,
"99.990000" : 57088
}
},
"lat_ns" : {
"min" : 5540,
"max" : 115927,
"mean" : 6300.778575,
"stddev" : 1106.768573,
"N" : 1072779
},
"bw_min" : 76414,
"bw_max" : 76744,
"bw_agg" : 100.000000,
"bw_mean" : 76630.071429,
"bw_dev" : 96.866652,
"bw_samples" : 14,
"iops_min" : 152828,
"iops_max" : 153488,
"iops_mean" : 153260.357143,
"iops_stddev" : 193.638683,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 21.700000,
"sys_cpu" : 78.285714,
"ctx" : 9,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.863552,
"10" : 98.266745,
"20" : 0.853577,
"50" : 0.010000,
"100" : 0.011466,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=5827: Wed Oct 26 15:03:17 2022
read: IOPS=153k, BW=74.8MiB/s (78.5MB/s)(524MiB/7001msec)
slat (nsec): min=1663, max=17694, avg=1826.52, stdev=127.59
clat (nsec): min=250, max=114004, avg=4474.26, stdev=1097.45
lat (usec): min=5, max=115, avg= 6.30, stdev= 1.11
clat percentiles (nsec):
| 1.00th=[ 4016], 5.00th=[ 4256], 10.00th=[ 4256], 20.00th=[ 4256],
| 30.00th=[ 4320], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4384],
| 70.00th=[ 4384], 80.00th=[ 4448], 90.00th=[ 4448], 95.00th=[ 4576],
| 99.00th=[ 9792], 99.50th=[10688], 99.90th=[15296], 99.95th=[15424],
| 99.99th=[57088]
bw ( KiB/s): min=76414, max=76744, per=100.00%, avg=76630.07, stdev=96.87, samples=14
iops : min=152828, max=153488, avg=153260.36, stdev=193.64, samples=14
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.86%, 10=98.27%, 20=0.85%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=21.70%, sys=78.29%, ctx=9, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1072779,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=74.8MiB/s (78.5MB/s), 74.8MiB/s-74.8MiB/s (78.5MB/s-78.5MB/s), io=524MiB (549MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789411,
"timestamp_ms" : 1666789411269,
"time" : "Wed Oct 26 15:03:31 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 390937600,
"io_kbytes" : 381775,
"bw_bytes" : 55840251,
"bw" : 54531,
"iops" : 109062.991001,
"runtime" : 7001,
"total_ios" : 763550,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1503,
"max" : 14387,
"mean" : 1667.139119,
"stddev" : 81.310916,
"N" : 763550
},
"clat_ns" : {
"min" : 230,
"max" : 118001,
"mean" : 7204.880431,
"stddev" : 960.561063,
"N" : 763550,
"percentile" : {
"1.000000" : 6752,
"5.000000" : 6944,
"10.000000" : 6944,
"20.000000" : 7008,
"30.000000" : 7008,
"40.000000" : 7072,
"50.000000" : 7136,
"60.000000" : 7136,
"70.000000" : 7200,
"80.000000" : 7200,
"90.000000" : 7264,
"95.000000" : 7392,
"99.000000" : 10176,
"99.500000" : 13504,
"99.900000" : 15552,
"99.950000" : 15808,
"99.990000" : 57600
}
},
"lat_ns" : {
"min" : 7204,
"max" : 119654,
"mean" : 8872.019549,
"stddev" : 965.822097,
"N" : 763550
},
"bw_min" : 54161,
"bw_max" : 54881,
"bw_agg" : 100.000000,
"bw_mean" : 54540.928571,
"bw_dev" : 272.108424,
"bw_samples" : 14,
"iops_min" : 108322,
"iops_max" : 109762,
"iops_mean" : 109081.928571,
"iops_stddev" : 544.151628,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 8.357143,
"sys_cpu" : 36.128571,
"ctx" : 763553,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 98.860847,
"20" : 1.121865,
"50" : 0.010000,
"100" : 0.013228,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=5874: Wed Oct 26 15:03:31 2022
read: IOPS=109k, BW=53.3MiB/s (55.8MB/s)(373MiB/7001msec)
slat (nsec): min=1503, max=14387, avg=1667.14, stdev=81.31
clat (nsec): min=230, max=118001, avg=7204.88, stdev=960.56
lat (nsec): min=7204, max=119654, avg=8872.02, stdev=965.82
clat percentiles (nsec):
| 1.00th=[ 6752], 5.00th=[ 6944], 10.00th=[ 6944], 20.00th=[ 7008],
| 30.00th=[ 7008], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7136],
| 70.00th=[ 7200], 80.00th=[ 7200], 90.00th=[ 7264], 95.00th=[ 7392],
| 99.00th=[10176], 99.50th=[13504], 99.90th=[15552], 99.95th=[15808],
| 99.99th=[57600]
bw ( KiB/s): min=54161, max=54881, per=100.00%, avg=54540.93, stdev=272.11, samples=14
iops : min=108322, max=109762, avg=109081.93, stdev=544.15, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=98.86%, 20=1.12%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%
cpu : usr=8.36%, sys=36.13%, ctx=763553, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=763550,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=53.3MiB/s (55.8MB/s), 53.3MiB/s-53.3MiB/s (55.8MB/s-55.8MB/s), io=373MiB (391MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789425,
"timestamp_ms" : 1666789425191,
"time" : "Wed Oct 26 15:03:45 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 390010880,
"io_kbytes" : 380870,
"bw_bytes" : 55715840,
"bw" : 54410,
"iops" : 108819.857143,
"runtime" : 7000,
"total_ios" : 761739,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1523,
"max" : 23895,
"mean" : 1676.788178,
"stddev" : 77.910712,
"N" : 761740
},
"clat_ns" : {
"min" : 211,
"max" : 134522,
"mean" : 7217.694233,
"stddev" : 1010.392200,
"N" : 761739,
"percentile" : {
"1.000000" : 6752,
"5.000000" : 6944,
"10.000000" : 7008,
"20.000000" : 7008,
"30.000000" : 7008,
"40.000000" : 7072,
"50.000000" : 7136,
"60.000000" : 7136,
"70.000000" : 7200,
"80.000000" : 7200,
"90.000000" : 7328,
"95.000000" : 7392,
"99.000000" : 10176,
"99.500000" : 13632,
"99.900000" : 15552,
"99.950000" : 15808,
"99.990000" : 59648
}
},
"lat_ns" : {
"min" : 7003,
"max" : 136165,
"mean" : 8894.482495,
"stddev" : 1014.288753,
"N" : 761739
},
"bw_min" : 54271,
"bw_max" : 54567,
"bw_agg" : 100.000000,
"bw_mean" : 54411.928571,
"bw_dev" : 78.668207,
"bw_samples" : 14,
"iops_min" : 108542,
"iops_max" : 109134,
"iops_mean" : 108823.928571,
"iops_stddev" : 157.274119,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 9.200000,
"sys_cpu" : 35.271429,
"ctx" : 761738,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 98.872448,
"20" : 1.107466,
"50" : 0.010000,
"100" : 0.015228,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=5922: Wed Oct 26 15:03:45 2022
read: IOPS=109k, BW=53.1MiB/s (55.7MB/s)(372MiB/7000msec)
slat (nsec): min=1523, max=23895, avg=1676.79, stdev=77.91
clat (nsec): min=211, max=134522, avg=7217.69, stdev=1010.39
lat (usec): min=7, max=136, avg= 8.89, stdev= 1.01
clat percentiles (nsec):
| 1.00th=[ 6752], 5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7008],
| 30.00th=[ 7008], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7136],
| 70.00th=[ 7200], 80.00th=[ 7200], 90.00th=[ 7328], 95.00th=[ 7392],
| 99.00th=[10176], 99.50th=[13632], 99.90th=[15552], 99.95th=[15808],
| 99.99th=[59648]
bw ( KiB/s): min=54271, max=54567, per=100.00%, avg=54411.93, stdev=78.67, samples=14
iops : min=108542, max=109134, avg=108823.93, stdev=157.27, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=98.87%, 20=1.11%, 50=0.01%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=9.20%, sys=35.27%, ctx=761738, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=761739,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=53.1MiB/s (55.7MB/s), 53.1MiB/s-53.1MiB/s (55.7MB/s-55.7MB/s), io=372MiB (390MB), run=7000-7000msec
$ rm /tmp/fio-output_512_1_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789439,
"timestamp_ms" : 1666789439029,
"time" : "Wed Oct 26 15:03:59 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 390489088,
"io_kbytes" : 381337,
"bw_bytes" : 55776187,
"bw" : 54468,
"iops" : 108937.723182,
"runtime" : 7001,
"total_ios" : 762673,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1512,
"max" : 22973,
"mean" : 1663.808307,
"stddev" : 81.901888,
"N" : 762674
},
"clat_ns" : {
"min" : 2965,
"max" : 339156,
"mean" : 7211.033603,
"stddev" : 1048.223194,
"N" : 762673,
"percentile" : {
"1.000000" : 6752,
"5.000000" : 6944,
"10.000000" : 6944,
"20.000000" : 7008,
"30.000000" : 7008,
"40.000000" : 7072,
"50.000000" : 7136,
"60.000000" : 7136,
"70.000000" : 7200,
"80.000000" : 7200,
"90.000000" : 7264,
"95.000000" : 7392,
"99.000000" : 10176,
"99.500000" : 13504,
"99.900000" : 15552,
"99.950000" : 15808,
"99.990000" : 57600
}
},
"lat_ns" : {
"min" : 7304,
"max" : 340799,
"mean" : 8874.841846,
"stddev" : 1052.530746,
"N" : 762673
},
"bw_min" : 54216,
"bw_max" : 54781,
"bw_agg" : 100.000000,
"bw_mean" : 54498.571429,
"bw_dev" : 162.638681,
"bw_samples" : 14,
"iops_min" : 108432,
"iops_max" : 109562,
"iops_mean" : 108997.285714,
"iops_stddev" : 325.114283,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 10.714286,
"sys_cpu" : 33.800000,
"ctx" : 762675,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 98.851539,
"20" : 1.126957,
"50" : 0.010000,
"100" : 0.014685,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=5970: Wed Oct 26 15:03:59 2022
read: IOPS=109k, BW=53.2MiB/s (55.8MB/s)(372MiB/7001msec)
slat (nsec): min=1512, max=22973, avg=1663.81, stdev=81.90
clat (usec): min=2, max=339, avg= 7.21, stdev= 1.05
lat (usec): min=7, max=340, avg= 8.87, stdev= 1.05
clat percentiles (nsec):
| 1.00th=[ 6752], 5.00th=[ 6944], 10.00th=[ 6944], 20.00th=[ 7008],
| 30.00th=[ 7008], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7136],
| 70.00th=[ 7200], 80.00th=[ 7200], 90.00th=[ 7264], 95.00th=[ 7392],
| 99.00th=[10176], 99.50th=[13504], 99.90th=[15552], 99.95th=[15808],
| 99.99th=[57600]
bw ( KiB/s): min=54216, max=54781, per=100.00%, avg=54498.57, stdev=162.64, samples=14
iops : min=108432, max=109562, avg=108997.29, stdev=325.11, samples=14
lat (usec) : 4=0.01%, 10=98.85%, 20=1.13%, 50=0.01%, 100=0.01%
lat (usec) : 500=0.01%
cpu : usr=10.71%, sys=33.80%, ctx=762675, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=762673,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=53.2MiB/s (55.8MB/s), 53.2MiB/s-53.2MiB/s (55.8MB/s-55.8MB/s), io=372MiB (390MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789452,
"timestamp_ms" : 1666789452865,
"time" : "Wed Oct 26 15:04:12 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 391555584,
"io_kbytes" : 382378,
"bw_bytes" : 55928522,
"bw" : 54617,
"iops" : 109235.394944,
"runtime" : 7001,
"total_ios" : 764757,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1653,
"max" : 6341,
"mean" : 1761.576344,
"stddev" : 74.909216,
"N" : 764757
},
"clat_ns" : {
"min" : 2775,
"max" : 94186,
"mean" : 7098.314393,
"stddev" : 981.590607,
"N" : 764757,
"percentile" : {
"1.000000" : 6624,
"5.000000" : 6816,
"10.000000" : 6880,
"20.000000" : 6880,
"30.000000" : 6944,
"40.000000" : 6944,
"50.000000" : 7008,
"60.000000" : 7072,
"70.000000" : 7072,
"80.000000" : 7072,
"90.000000" : 7200,
"95.000000" : 7264,
"99.000000" : 10176,
"99.500000" : 13504,
"99.900000" : 15424,
"99.950000" : 15680,
"99.990000" : 58112
}
},
"lat_ns" : {
"min" : 7384,
"max" : 95980,
"mean" : 8859.890737,
"stddev" : 985.744709,
"N" : 764757
},
"bw_min" : 54489,
"bw_max" : 54838,
"bw_agg" : 100.000000,
"bw_mean" : 54647.692308,
"bw_dev" : 106.501475,
"bw_samples" : 13,
"iops_min" : 108978,
"iops_max" : 109676,
"iops_mean" : 109295.461538,
"iops_stddev" : 213.112887,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.628571,
"sys_cpu" : 27.585714,
"ctx" : 764759,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 98.872583,
"20" : 1.104534,
"50" : 0.010000,
"100" : 0.014253,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 1067722,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 5364,
"write_ticks" : 0,
"in_queue" : 5364,
"util" : 99.045298
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=6017: Wed Oct 26 15:04:12 2022
read: IOPS=109k, BW=53.3MiB/s (55.9MB/s)(373MiB/7001msec)
slat (nsec): min=1653, max=6341, avg=1761.58, stdev=74.91
clat (nsec): min=2775, max=94186, avg=7098.31, stdev=981.59
lat (nsec): min=7384, max=95980, avg=8859.89, stdev=985.74
clat percentiles (nsec):
| 1.00th=[ 6624], 5.00th=[ 6816], 10.00th=[ 6880], 20.00th=[ 6880],
| 30.00th=[ 6944], 40.00th=[ 6944], 50.00th=[ 7008], 60.00th=[ 7072],
| 70.00th=[ 7072], 80.00th=[ 7072], 90.00th=[ 7200], 95.00th=[ 7264],
| 99.00th=[10176], 99.50th=[13504], 99.90th=[15424], 99.95th=[15680],
| 99.99th=[58112]
bw ( KiB/s): min=54489, max=54838, per=100.00%, avg=54647.69, stdev=106.50, samples=13
iops : min=108978, max=109676, avg=109295.46, stdev=213.11, samples=13
lat (usec) : 4=0.01%, 10=98.87%, 20=1.10%, 50=0.01%, 100=0.01%
cpu : usr=15.63%, sys=27.59%, ctx=764759, majf=0, minf=1
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=764757,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=53.3MiB/s (55.9MB/s), 53.3MiB/s-53.3MiB/s (55.9MB/s-55.9MB/s), io=373MiB (392MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=1067722/0, merge=0/0, ticks=5364/0, in_queue=5364, util=99.05%
$ rm /tmp/fio-output_512_1_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789466,
"timestamp_ms" : 1666789466728,
"time" : "Wed Oct 26 15:04:26 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 393853440,
"io_kbytes" : 384622,
"bw_bytes" : 56256740,
"bw" : 54938,
"iops" : 109876.446222,
"runtime" : 7001,
"total_ios" : 769245,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1633,
"max" : 15660,
"mean" : 1733.714902,
"stddev" : 59.717360,
"N" : 769245
},
"clat_ns" : {
"min" : 581,
"max" : 88846,
"mean" : 7084.517244,
"stddev" : 968.322335,
"N" : 769245,
"percentile" : {
"1.000000" : 6624,
"5.000000" : 6816,
"10.000000" : 6816,
"20.000000" : 6880,
"30.000000" : 6880,
"40.000000" : 6944,
"50.000000" : 7008,
"60.000000" : 7008,
"70.000000" : 7072,
"80.000000" : 7072,
"90.000000" : 7200,
"95.000000" : 7264,
"99.000000" : 10176,
"99.500000" : 13504,
"99.900000" : 15424,
"99.950000" : 15680,
"99.990000" : 58112
}
},
"lat_ns" : {
"min" : 7294,
"max" : 90539,
"mean" : 8818.232146,
"stddev" : 971.397702,
"N" : 769245
},
"bw_min" : 54727,
"bw_max" : 55130,
"bw_agg" : 100.000000,
"bw_mean" : 54952.357143,
"bw_dev" : 130.746147,
"bw_samples" : 14,
"iops_min" : 109454,
"iops_max" : 110260,
"iops_mean" : 109904.714286,
"iops_stddev" : 261.492294,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.442857,
"sys_cpu" : 25.171429,
"ctx" : 769247,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 98.809872,
"20" : 1.172708,
"50" : 0.010000,
"100" : 0.013650,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 1099355,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 5515,
"write_ticks" : 0,
"in_queue" : 5515,
"util" : 99.069123
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=6063: Wed Oct 26 15:04:26 2022
read: IOPS=110k, BW=53.7MiB/s (56.3MB/s)(376MiB/7001msec)
slat (nsec): min=1633, max=15660, avg=1733.71, stdev=59.72
clat (nsec): min=581, max=88846, avg=7084.52, stdev=968.32
lat (nsec): min=7294, max=90539, avg=8818.23, stdev=971.40
clat percentiles (nsec):
| 1.00th=[ 6624], 5.00th=[ 6816], 10.00th=[ 6816], 20.00th=[ 6880],
| 30.00th=[ 6880], 40.00th=[ 6944], 50.00th=[ 7008], 60.00th=[ 7008],
| 70.00th=[ 7072], 80.00th=[ 7072], 90.00th=[ 7200], 95.00th=[ 7264],
| 99.00th=[10176], 99.50th=[13504], 99.90th=[15424], 99.95th=[15680],
| 99.99th=[58112]
bw ( KiB/s): min=54727, max=55130, per=100.00%, avg=54952.36, stdev=130.75, samples=14
iops : min=109454, max=110260, avg=109904.71, stdev=261.49, samples=14
lat (nsec) : 750=0.01%
lat (usec) : 4=0.01%, 10=98.81%, 20=1.17%, 50=0.01%, 100=0.01%
cpu : usr=17.44%, sys=25.17%, ctx=769247, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=769245,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=53.7MiB/s (56.3MB/s), 53.7MiB/s-53.7MiB/s (56.3MB/s-56.3MB/s), io=376MiB (394MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=1099355/0, merge=0/0, ticks=5515/0, in_queue=5515, util=99.07%
$ rm /tmp/fio-output_512_1_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789480,
"timestamp_ms" : 1666789480604,
"time" : "Wed Oct 26 15:04:40 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 392696320,
"io_kbytes" : 383492,
"bw_bytes" : 56099474,
"bw" : 54784,
"iops" : 109569.285714,
"runtime" : 7000,
"total_ios" : 766985,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1613,
"max" : 9728,
"mean" : 1717.578988,
"stddev" : 59.288933,
"N" : 766985
},
"clat_ns" : {
"min" : 471,
"max" : 114094,
"mean" : 7108.687395,
"stddev" : 1019.143377,
"N" : 766985,
"percentile" : {
"1.000000" : 6624,
"5.000000" : 6816,
"10.000000" : 6880,
"20.000000" : 6880,
"30.000000" : 6944,
"40.000000" : 6944,
"50.000000" : 7008,
"60.000000" : 7072,
"70.000000" : 7072,
"80.000000" : 7136,
"90.000000" : 7200,
"95.000000" : 7328,
"99.000000" : 10176,
"99.500000" : 13504,
"99.900000" : 15552,
"99.950000" : 15680,
"99.990000" : 60160
}
},
"lat_ns" : {
"min" : 6893,
"max" : 115777,
"mean" : 8826.266383,
"stddev" : 1021.839338,
"N" : 766985
},
"bw_min" : 54634,
"bw_max" : 55015,
"bw_agg" : 100.000000,
"bw_mean" : 54791.000000,
"bw_dev" : 131.780704,
"bw_samples" : 14,
"iops_min" : 109268,
"iops_max" : 110030,
"iops_mean" : 109582.071429,
"iops_stddev" : 263.670092,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 19.400000,
"sys_cpu" : 23.571429,
"ctx" : 766986,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 98.838439,
"20" : 1.141222,
"50" : 0.010000,
"100" : 0.016037,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 1099551,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 5504,
"write_ticks" : 0,
"in_queue" : 5504,
"util" : 99.069123
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=6110: Wed Oct 26 15:04:40 2022
read: IOPS=110k, BW=53.5MiB/s (56.1MB/s)(375MiB/7000msec)
slat (nsec): min=1613, max=9728, avg=1717.58, stdev=59.29
clat (nsec): min=471, max=114094, avg=7108.69, stdev=1019.14
lat (usec): min=6, max=115, avg= 8.83, stdev= 1.02
clat percentiles (nsec):
| 1.00th=[ 6624], 5.00th=[ 6816], 10.00th=[ 6880], 20.00th=[ 6880],
| 30.00th=[ 6944], 40.00th=[ 6944], 50.00th=[ 7008], 60.00th=[ 7072],
| 70.00th=[ 7072], 80.00th=[ 7136], 90.00th=[ 7200], 95.00th=[ 7328],
| 99.00th=[10176], 99.50th=[13504], 99.90th=[15552], 99.95th=[15680],
| 99.99th=[60160]
bw ( KiB/s): min=54634, max=55015, per=100.00%, avg=54791.00, stdev=131.78, samples=14
iops : min=109268, max=110030, avg=109582.07, stdev=263.67, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=98.84%, 20=1.14%, 50=0.01%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=19.40%, sys=23.57%, ctx=766986, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=766985,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=53.5MiB/s (56.1MB/s), 53.5MiB/s-53.5MiB/s (56.1MB/s-56.1MB/s), io=375MiB (393MB), run=7000-7000msec
Disk stats (read/write):
nvme0n1: ios=1099551/0, merge=0/0, ticks=5504/0, in_queue=5504, util=99.07%
$ rm /tmp/fio-output_512_1_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-bdev_aio_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789495,
"timestamp_ms" : 1666789495981,
"time" : "Wed Oct 26 15:04:55 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 542872576,
"io_kbytes" : 530149,
"bw_bytes" : 77553225,
"bw" : 75735,
"iops" : 151471.000000,
"runtime" : 7000,
"total_ios" : 1060297,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1733,
"max" : 17563,
"mean" : 1907.857454,
"stddev" : 129.896684,
"N" : 1060298
},
"clat_ns" : {
"min" : 230,
"max" : 96982,
"mean" : 4470.611756,
"stddev" : 1041.750438,
"N" : 1060297,
"percentile" : {
"1.000000" : 4048,
"5.000000" : 4256,
"10.000000" : 4256,
"20.000000" : 4256,
"30.000000" : 4320,
"40.000000" : 4320,
"50.000000" : 4320,
"60.000000" : 4384,
"70.000000" : 4384,
"80.000000" : 4448,
"90.000000" : 4448,
"95.000000" : 4640,
"99.000000" : 9664,
"99.500000" : 10688,
"99.900000" : 15296,
"99.950000" : 15424,
"99.990000" : 49408
}
},
"lat_ns" : {
"min" : 5640,
"max" : 98935,
"mean" : 6378.469139,
"stddev" : 1051.660338,
"N" : 1060297
},
"bw_min" : 75451,
"bw_max" : 76128,
"bw_agg" : 100.000000,
"bw_mean" : 75738.142857,
"bw_dev" : 208.255707,
"bw_samples" : 14,
"iops_min" : 150902,
"iops_max" : 152256,
"iops_mean" : 151476.285714,
"iops_stddev" : 416.511414,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 77.700000,
"sys_cpu" : 22.271429,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.755637,
"10" : 98.431383,
"20" : 0.798361,
"50" : 0.010000,
"100" : 0.010000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=6168: Wed Oct 26 15:04:55 2022
read: IOPS=151k, BW=74.0MiB/s (77.6MB/s)(518MiB/7000msec)
slat (nsec): min=1733, max=17563, avg=1907.86, stdev=129.90
clat (nsec): min=230, max=96982, avg=4470.61, stdev=1041.75
lat (nsec): min=5640, max=98935, avg=6378.47, stdev=1051.66
clat percentiles (nsec):
| 1.00th=[ 4048], 5.00th=[ 4256], 10.00th=[ 4256], 20.00th=[ 4256],
| 30.00th=[ 4320], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4384],
| 70.00th=[ 4384], 80.00th=[ 4448], 90.00th=[ 4448], 95.00th=[ 4640],
| 99.00th=[ 9664], 99.50th=[10688], 99.90th=[15296], 99.95th=[15424],
| 99.99th=[49408]
bw ( KiB/s): min=75451, max=76128, per=100.00%, avg=75738.14, stdev=208.26, samples=14
iops : min=150902, max=152256, avg=151476.29, stdev=416.51, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 2=0.01%, 4=0.76%, 10=98.43%, 20=0.80%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=77.70%, sys=22.27%, ctx=10, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1060297,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=74.0MiB/s (77.6MB/s), 74.0MiB/s-74.0MiB/s (77.6MB/s-77.6MB/s), io=518MiB (543MB), run=7000-7000msec
$ rm /tmp/fio-output_512_1_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-bdev_aio_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789511,
"timestamp_ms" : 1666789511342,
"time" : "Wed Oct 26 15:05:11 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 544499200,
"io_kbytes" : 531737,
"bw_bytes" : 77774489,
"bw" : 75951,
"iops" : 151903.299529,
"runtime" : 7001,
"total_ios" : 1063475,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1753,
"max" : 16831,
"mean" : 1900.398060,
"stddev" : 139.445188,
"N" : 1063475
},
"clat_ns" : {
"min" : 210,
"max" : 77395,
"mean" : 4459.766281,
"stddev" : 1077.834294,
"N" : 1063475,
"percentile" : {
"1.000000" : 4048,
"5.000000" : 4192,
"10.000000" : 4256,
"20.000000" : 4256,
"30.000000" : 4256,
"40.000000" : 4320,
"50.000000" : 4320,
"60.000000" : 4384,
"70.000000" : 4384,
"80.000000" : 4384,
"90.000000" : 4448,
"95.000000" : 4576,
"99.000000" : 9664,
"99.500000" : 10560,
"99.900000" : 15296,
"99.950000" : 15424,
"99.990000" : 57088
}
},
"lat_ns" : {
"min" : 5620,
"max" : 79289,
"mean" : 6360.164341,
"stddev" : 1089.019246,
"N" : 1063475
},
"bw_min" : 75781,
"bw_max" : 76168,
"bw_agg" : 100.000000,
"bw_mean" : 75991.692308,
"bw_dev" : 122.716058,
"bw_samples" : 13,
"iops_min" : 151562,
"iops_max" : 152336,
"iops_mean" : 151983.461538,
"iops_stddev" : 245.379507,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 74.900000,
"sys_cpu" : 25.071429,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.799643,
"10" : 98.374292,
"20" : 0.809046,
"50" : 0.010000,
"100" : 0.012130,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=6226: Wed Oct 26 15:05:11 2022
read: IOPS=152k, BW=74.2MiB/s (77.8MB/s)(519MiB/7001msec)
slat (nsec): min=1753, max=16831, avg=1900.40, stdev=139.45
clat (nsec): min=210, max=77395, avg=4459.77, stdev=1077.83
lat (nsec): min=5620, max=79289, avg=6360.16, stdev=1089.02
clat percentiles (nsec):
| 1.00th=[ 4048], 5.00th=[ 4192], 10.00th=[ 4256], 20.00th=[ 4256],
| 30.00th=[ 4256], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4384],
| 70.00th=[ 4384], 80.00th=[ 4384], 90.00th=[ 4448], 95.00th=[ 4576],
| 99.00th=[ 9664], 99.50th=[10560], 99.90th=[15296], 99.95th=[15424],
| 99.99th=[57088]
bw ( KiB/s): min=75781, max=76168, per=100.00%, avg=75991.69, stdev=122.72, samples=13
iops : min=151562, max=152336, avg=151983.46, stdev=245.38, samples=13
lat (nsec) : 250=0.01%, 500=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.80%, 10=98.37%, 20=0.81%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=74.90%, sys=25.07%, ctx=10, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1063475,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=74.2MiB/s (77.8MB/s), 74.2MiB/s-74.2MiB/s (77.8MB/s-77.8MB/s), io=519MiB (544MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-bdev_aio_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789526,
"timestamp_ms" : 1666789526714,
"time" : "Wed Oct 26 15:05:26 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 542993920,
"io_kbytes" : 530267,
"bw_bytes" : 77559480,
"bw" : 75741,
"iops" : 151483.359520,
"runtime" : 7001,
"total_ios" : 1060535,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1733,
"max" : 17993,
"mean" : 1902.296994,
"stddev" : 142.855709,
"N" : 1060535
},
"clat_ns" : {
"min" : 220,
"max" : 99156,
"mean" : 4465.624404,
"stddev" : 1078.405463,
"N" : 1060535,
"percentile" : {
"1.000000" : 4048,
"5.000000" : 4256,
"10.000000" : 4256,
"20.000000" : 4256,
"30.000000" : 4320,
"40.000000" : 4320,
"50.000000" : 4320,
"60.000000" : 4384,
"70.000000" : 4384,
"80.000000" : 4384,
"90.000000" : 4448,
"95.000000" : 4576,
"99.000000" : 9664,
"99.500000" : 10688,
"99.900000" : 15296,
"99.950000" : 15424,
"99.990000" : 57088
}
},
"lat_ns" : {
"min" : 5630,
"max" : 101070,
"mean" : 6367.921398,
"stddev" : 1089.091086,
"N" : 1060535
},
"bw_min" : 75607,
"bw_max" : 75920,
"bw_agg" : 100.000000,
"bw_mean" : 75783.928571,
"bw_dev" : 71.611628,
"bw_samples" : 14,
"iops_min" : 151214,
"iops_max" : 151838,
"iops_mean" : 151567.857143,
"iops_stddev" : 142.936703,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 76.042857,
"sys_cpu" : 23.928571,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.758202,
"10" : 98.419666,
"20" : 0.804028,
"50" : 0.010000,
"100" : 0.011881,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=6285: Wed Oct 26 15:05:26 2022
read: IOPS=151k, BW=74.0MiB/s (77.6MB/s)(518MiB/7001msec)
slat (nsec): min=1733, max=17993, avg=1902.30, stdev=142.86
clat (nsec): min=220, max=99156, avg=4465.62, stdev=1078.41
lat (usec): min=5, max=101, avg= 6.37, stdev= 1.09
clat percentiles (nsec):
| 1.00th=[ 4048], 5.00th=[ 4256], 10.00th=[ 4256], 20.00th=[ 4256],
| 30.00th=[ 4320], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4384],
| 70.00th=[ 4384], 80.00th=[ 4384], 90.00th=[ 4448], 95.00th=[ 4576],
| 99.00th=[ 9664], 99.50th=[10688], 99.90th=[15296], 99.95th=[15424],
| 99.99th=[57088]
bw ( KiB/s): min=75607, max=75920, per=100.00%, avg=75783.93, stdev=71.61, samples=14
iops : min=151214, max=151838, avg=151567.86, stdev=142.94, samples=14
lat (nsec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.76%, 10=98.42%, 20=0.80%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=76.04%, sys=23.93%, ctx=10, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1060535,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=74.0MiB/s (77.6MB/s), 74.0MiB/s-74.0MiB/s (77.6MB/s-77.6MB/s), io=518MiB (543MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789542,
"timestamp_ms" : 1666789542121,
"time" : "Wed Oct 26 15:05:42 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 523051520,
"io_kbytes" : 510792,
"bw_bytes" : 74710972,
"bw" : 72959,
"iops" : 145919.868590,
"runtime" : 7001,
"total_ios" : 1021585,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1753,
"max" : 40205,
"mean" : 1902.474167,
"stddev" : 112.382836,
"N" : 1021585
},
"clat_ns" : {
"min" : 591,
"max" : 101370,
"mean" : 4716.904850,
"stddev" : 1098.648880,
"N" : 1021585,
"percentile" : {
"1.000000" : 4384,
"5.000000" : 4384,
"10.000000" : 4448,
"20.000000" : 4448,
"30.000000" : 4448,
"40.000000" : 4640,
"50.000000" : 4640,
"60.000000" : 4640,
"70.000000" : 4704,
"80.000000" : 4704,
"90.000000" : 4704,
"95.000000" : 4896,
"99.000000" : 9664,
"99.500000" : 10944,
"99.900000" : 15296,
"99.950000" : 15424,
"99.990000" : 58624
}
},
"lat_ns" : {
"min" : 5871,
"max" : 103304,
"mean" : 6619.379017,
"stddev" : 1103.785829,
"N" : 1021585
},
"bw_min" : 72756,
"bw_max" : 73193,
"bw_agg" : 100.000000,
"bw_mean" : 72972.928571,
"bw_dev" : 114.125750,
"bw_samples" : 14,
"iops_min" : 145512,
"iops_max" : 146386,
"iops_mean" : 145945.857143,
"iops_stddev" : 228.251500,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 54.728571,
"sys_cpu" : 45.242857,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.017913,
"10" : 99.114122,
"20" : 0.844962,
"50" : 0.010000,
"100" : 0.014389,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=6343: Wed Oct 26 15:05:42 2022
read: IOPS=146k, BW=71.2MiB/s (74.7MB/s)(499MiB/7001msec)
slat (nsec): min=1753, max=40205, avg=1902.47, stdev=112.38
clat (nsec): min=591, max=101370, avg=4716.90, stdev=1098.65
lat (usec): min=5, max=103, avg= 6.62, stdev= 1.10
clat percentiles (nsec):
| 1.00th=[ 4384], 5.00th=[ 4384], 10.00th=[ 4448], 20.00th=[ 4448],
| 30.00th=[ 4448], 40.00th=[ 4640], 50.00th=[ 4640], 60.00th=[ 4640],
| 70.00th=[ 4704], 80.00th=[ 4704], 90.00th=[ 4704], 95.00th=[ 4896],
| 99.00th=[ 9664], 99.50th=[10944], 99.90th=[15296], 99.95th=[15424],
| 99.99th=[58624]
bw ( KiB/s): min=72756, max=73193, per=100.00%, avg=72972.93, stdev=114.13, samples=14
iops : min=145512, max=146386, avg=145945.86, stdev=228.25, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.02%, 10=99.11%, 20=0.84%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=54.73%, sys=45.24%, ctx=10, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1021585,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=71.2MiB/s (74.7MB/s), 71.2MiB/s-71.2MiB/s (74.7MB/s-74.7MB/s), io=499MiB (523MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789557,
"timestamp_ms" : 1666789557225,
"time" : "Wed Oct 26 15:05:57 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 522268672,
"io_kbytes" : 510028,
"bw_bytes" : 74609810,
"bw" : 72861,
"iops" : 145722.142857,
"runtime" : 7000,
"total_ios" : 1020055,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1773,
"max" : 14247,
"mean" : 1911.317889,
"stddev" : 104.278118,
"N" : 1020056
},
"clat_ns" : {
"min" : 601,
"max" : 117801,
"mean" : 4724.615978,
"stddev" : 1066.997954,
"N" : 1020055,
"percentile" : {
"1.000000" : 4256,
"5.000000" : 4448,
"10.000000" : 4448,
"20.000000" : 4448,
"30.000000" : 4512,
"40.000000" : 4512,
"50.000000" : 4640,
"60.000000" : 4704,
"70.000000" : 4704,
"80.000000" : 4704,
"90.000000" : 4768,
"95.000000" : 4768,
"99.000000" : 9664,
"99.500000" : 10944,
"99.900000" : 15296,
"99.950000" : 15424,
"99.990000" : 57088
}
},
"lat_ns" : {
"min" : 5851,
"max" : 119655,
"mean" : 6635.933895,
"stddev" : 1071.288496,
"N" : 1020055
},
"bw_min" : 72757,
"bw_max" : 73024,
"bw_agg" : 100.000000,
"bw_mean" : 72863.785714,
"bw_dev" : 78.671839,
"bw_samples" : 14,
"iops_min" : 145514,
"iops_max" : 146048,
"iops_mean" : 145727.642857,
"iops_stddev" : 157.314779,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 48.328571,
"sys_cpu" : 51.642857,
"ctx" : 14,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.014607,
"10" : 99.142791,
"20" : 0.821034,
"50" : 0.010000,
"100" : 0.012646,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=6400: Wed Oct 26 15:05:57 2022
read: IOPS=146k, BW=71.2MiB/s (74.6MB/s)(498MiB/7000msec)
slat (nsec): min=1773, max=14247, avg=1911.32, stdev=104.28
clat (nsec): min=601, max=117801, avg=4724.62, stdev=1067.00
lat (usec): min=5, max=119, avg= 6.64, stdev= 1.07
clat percentiles (nsec):
| 1.00th=[ 4256], 5.00th=[ 4448], 10.00th=[ 4448], 20.00th=[ 4448],
| 30.00th=[ 4512], 40.00th=[ 4512], 50.00th=[ 4640], 60.00th=[ 4704],
| 70.00th=[ 4704], 80.00th=[ 4704], 90.00th=[ 4768], 95.00th=[ 4768],
| 99.00th=[ 9664], 99.50th=[10944], 99.90th=[15296], 99.95th=[15424],
| 99.99th=[57088]
bw ( KiB/s): min=72757, max=73024, per=100.00%, avg=72863.79, stdev=78.67, samples=14
iops : min=145514, max=146048, avg=145727.64, stdev=157.31, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=99.14%, 20=0.82%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=48.33%, sys=51.64%, ctx=14, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1020055,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=71.2MiB/s (74.6MB/s), 71.2MiB/s-71.2MiB/s (74.6MB/s-74.6MB/s), io=498MiB (522MB), run=7000-7000msec
$ rm /tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789572,
"timestamp_ms" : 1666789572451,
"time" : "Wed Oct 26 15:06:12 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 521175040,
"io_kbytes" : 508960,
"bw_bytes" : 74442942,
"bw" : 72698,
"iops" : 145396.371947,
"runtime" : 7001,
"total_ios" : 1017920,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1743,
"max" : 14618,
"mean" : 1896.726667,
"stddev" : 103.752460,
"N" : 1017920
},
"clat_ns" : {
"min" : 601,
"max" : 117551,
"mean" : 4735.870113,
"stddev" : 1060.455946,
"N" : 1017920,
"percentile" : {
"1.000000" : 4384,
"5.000000" : 4384,
"10.000000" : 4448,
"20.000000" : 4448,
"30.000000" : 4640,
"40.000000" : 4640,
"50.000000" : 4640,
"60.000000" : 4640,
"70.000000" : 4704,
"80.000000" : 4704,
"90.000000" : 4704,
"95.000000" : 4896,
"99.000000" : 9792,
"99.500000" : 10816,
"99.900000" : 15168,
"99.950000" : 15424,
"99.990000" : 56576
}
},
"lat_ns" : {
"min" : 5891,
"max" : 119544,
"mean" : 6632.596780,
"stddev" : 1064.048858,
"N" : 1017920
},
"bw_min" : 72585,
"bw_max" : 72865,
"bw_agg" : 100.000000,
"bw_mean" : 72711.142857,
"bw_dev" : 64.441813,
"bw_samples" : 14,
"iops_min" : 145170,
"iops_max" : 145730,
"iops_mean" : 145422.285714,
"iops_stddev" : 128.883626,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 53.528571,
"sys_cpu" : 46.457143,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.016111,
"10" : 99.099733,
"20" : 0.863329,
"50" : 0.010000,
"100" : 0.011199,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=6458: Wed Oct 26 15:06:12 2022
read: IOPS=145k, BW=71.0MiB/s (74.4MB/s)(497MiB/7001msec)
slat (nsec): min=1743, max=14618, avg=1896.73, stdev=103.75
clat (nsec): min=601, max=117551, avg=4735.87, stdev=1060.46
lat (usec): min=5, max=119, avg= 6.63, stdev= 1.06
clat percentiles (nsec):
| 1.00th=[ 4384], 5.00th=[ 4384], 10.00th=[ 4448], 20.00th=[ 4448],
| 30.00th=[ 4640], 40.00th=[ 4640], 50.00th=[ 4640], 60.00th=[ 4640],
| 70.00th=[ 4704], 80.00th=[ 4704], 90.00th=[ 4704], 95.00th=[ 4896],
| 99.00th=[ 9792], 99.50th=[10816], 99.90th=[15168], 99.95th=[15424],
| 99.99th=[56576]
bw ( KiB/s): min=72585, max=72865, per=100.00%, avg=72711.14, stdev=64.44, samples=14
iops : min=145170, max=145730, avg=145422.29, stdev=128.88, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.02%, 10=99.10%, 20=0.86%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=53.53%, sys=46.46%, ctx=10, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1017920,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=71.0MiB/s (74.4MB/s), 71.0MiB/s-71.0MiB/s (74.4MB/s-74.4MB/s), io=497MiB (521MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789586,
"timestamp_ms" : 1666789586262,
"time" : "Wed Oct 26 15:06:26 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 384857088,
"io_kbytes" : 375837,
"bw_bytes" : 54979584,
"bw" : 53691,
"iops" : 107381.857143,
"runtime" : 7000,
"total_ios" : 751673,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1693,
"max" : 46567,
"mean" : 1810.359939,
"stddev" : 98.872126,
"N" : 751674
},
"clat_ns" : {
"min" : 612,
"max" : 75983,
"mean" : 7221.342058,
"stddev" : 966.759816,
"N" : 751673,
"percentile" : {
"1.000000" : 6752,
"5.000000" : 6944,
"10.000000" : 7008,
"20.000000" : 7008,
"30.000000" : 7072,
"40.000000" : 7072,
"50.000000" : 7136,
"60.000000" : 7200,
"70.000000" : 7200,
"80.000000" : 7200,
"90.000000" : 7264,
"95.000000" : 7392,
"99.000000" : 10048,
"99.500000" : 13632,
"99.900000" : 15424,
"99.950000" : 15808,
"99.990000" : 57600
}
},
"lat_ns" : {
"min" : 7604,
"max" : 77766,
"mean" : 9031.702033,
"stddev" : 973.215408,
"N" : 751673
},
"bw_min" : 53585,
"bw_max" : 53789,
"bw_agg" : 100.000000,
"bw_mean" : 53692.785714,
"bw_dev" : 71.730455,
"bw_samples" : 14,
"iops_min" : 107170,
"iops_max" : 107578,
"iops_mean" : 107385.642857,
"iops_stddev" : 143.549298,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 11.500000,
"sys_cpu" : 35.457143,
"ctx" : 751676,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 98.988124,
"20" : 0.992320,
"50" : 0.010000,
"100" : 0.014767,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=6506: Wed Oct 26 15:06:26 2022
read: IOPS=107k, BW=52.4MiB/s (55.0MB/s)(367MiB/7000msec)
slat (nsec): min=1693, max=46567, avg=1810.36, stdev=98.87
clat (nsec): min=612, max=75983, avg=7221.34, stdev=966.76
lat (nsec): min=7604, max=77766, avg=9031.70, stdev=973.22
clat percentiles (nsec):
| 1.00th=[ 6752], 5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7008],
| 30.00th=[ 7072], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7200],
| 70.00th=[ 7200], 80.00th=[ 7200], 90.00th=[ 7264], 95.00th=[ 7392],
| 99.00th=[10048], 99.50th=[13632], 99.90th=[15424], 99.95th=[15808],
| 99.99th=[57600]
bw ( KiB/s): min=53585, max=53789, per=100.00%, avg=53692.79, stdev=71.73, samples=14
iops : min=107170, max=107578, avg=107385.64, stdev=143.55, samples=14
lat (nsec) : 750=0.01%
lat (usec) : 4=0.01%, 10=98.99%, 20=0.99%, 50=0.01%, 100=0.01%
cpu : usr=11.50%, sys=35.46%, ctx=751676, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=751673,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=52.4MiB/s (55.0MB/s), 52.4MiB/s-52.4MiB/s (55.0MB/s-55.0MB/s), io=367MiB (385MB), run=7000-7000msec
$ rm /tmp/fio-output_512_1_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789600,
"timestamp_ms" : 1666789600159,
"time" : "Wed Oct 26 15:06:40 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 385519104,
"io_kbytes" : 376483,
"bw_bytes" : 55066291,
"bw" : 53775,
"iops" : 107551.349807,
"runtime" : 7001,
"total_ios" : 752967,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1693,
"max" : 22682,
"mean" : 1809.645753,
"stddev" : 88.186978,
"N" : 752967
},
"clat_ns" : {
"min" : 672,
"max" : 75631,
"mean" : 7207.420906,
"stddev" : 1021.305121,
"N" : 752967,
"percentile" : {
"1.000000" : 6752,
"5.000000" : 6944,
"10.000000" : 6944,
"20.000000" : 7008,
"30.000000" : 7008,
"40.000000" : 7072,
"50.000000" : 7136,
"60.000000" : 7136,
"70.000000" : 7200,
"80.000000" : 7200,
"90.000000" : 7264,
"95.000000" : 7392,
"99.000000" : 10048,
"99.500000" : 13504,
"99.900000" : 15424,
"99.950000" : 15808,
"99.990000" : 60160
}
},
"lat_ns" : {
"min" : 7604,
"max" : 77385,
"mean" : 9017.066659,
"stddev" : 1026.795108,
"N" : 752967
},
"bw_min" : 53671,
"bw_max" : 53904,
"bw_agg" : 100.000000,
"bw_mean" : 53804.928571,
"bw_dev" : 70.010400,
"bw_samples" : 14,
"iops_min" : 107342,
"iops_max" : 107808,
"iops_mean" : 107610.000000,
"iops_stddev" : 139.982966,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 11.300000,
"sys_cpu" : 35.500000,
"ctx" : 752967,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 98.972863,
"20" : 1.005755,
"50" : 0.010000,
"100" : 0.017265,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=6554: Wed Oct 26 15:06:40 2022
read: IOPS=108k, BW=52.5MiB/s (55.1MB/s)(368MiB/7001msec)
slat (nsec): min=1693, max=22682, avg=1809.65, stdev=88.19
clat (nsec): min=672, max=75631, avg=7207.42, stdev=1021.31
lat (nsec): min=7604, max=77385, avg=9017.07, stdev=1026.80
clat percentiles (nsec):
| 1.00th=[ 6752], 5.00th=[ 6944], 10.00th=[ 6944], 20.00th=[ 7008],
| 30.00th=[ 7008], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7136],
| 70.00th=[ 7200], 80.00th=[ 7200], 90.00th=[ 7264], 95.00th=[ 7392],
| 99.00th=[10048], 99.50th=[13504], 99.90th=[15424], 99.95th=[15808],
| 99.99th=[60160]
bw ( KiB/s): min=53671, max=53904, per=100.00%, avg=53804.93, stdev=70.01, samples=14
iops : min=107342, max=107808, avg=107610.00, stdev=139.98, samples=14
lat (nsec) : 750=0.01%
lat (usec) : 4=0.01%, 10=98.97%, 20=1.01%, 50=0.01%, 100=0.02%
cpu : usr=11.30%, sys=35.50%, ctx=752967, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=752967,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=52.5MiB/s (55.1MB/s), 52.5MiB/s-52.5MiB/s (55.1MB/s-55.1MB/s), io=368MiB (386MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666789614,
"timestamp_ms" : 1666789614006,
"time" : "Wed Oct 26 15:06:54 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 382008832,
"io_kbytes" : 373055,
"bw_bytes" : 54564895,
"bw" : 53286,
"iops" : 106571.918297,
"runtime" : 7001,
"total_ios" : 746110,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1743,
"max" : 20578,
"mean" : 1853.316921,
"stddev" : 84.943754,
"N" : 746111
},
"clat_ns" : {
"min" : 571,
"max" : 462377,
"mean" : 7237.205919,
"stddev" : 1091.433101,
"N" : 746110,
"percentile" : {
"1.000000" : 6816,
"5.000000" : 6944,
"10.000000" : 7008,
"20.000000" : 7072,
"30.000000" : 7072,
"40.000000" : 7072,
"50.000000" : 7136,
"60.000000" : 7200,
"70.000000" : 7200,
"80.000000" : 7264,
"90.000000" : 7328,
"95.000000" : 7456,
"99.000000" : 9920,
"99.500000" : 13632,
"99.900000" : 15424,
"99.950000" : 16064,
"99.990000" : 57600
}
},
"lat_ns" : {
"min" : 6503,
"max" : 464231,
"mean" : 9090.522852,
"stddev" : 1096.371180,
"N" : 746110
},
"bw_min" : 53220,
"bw_max" : 53390,
"bw_agg" : 100.000000,
"bw_mean" : 53313.285714,
"bw_dev" : 55.447034,
"bw_samples" : 14,
"iops_min" : 106440,
"iops_max" : 106780,
"iops_mean" : 106626.714286,
"iops_stddev" : 111.025589,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.171429,
"sys_cpu" : 34.214286,
"ctx" : 746110,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 99.015829,
"20" : 0.965541,
"50" : 0.010000,
"100" : 0.013403,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=6602: Wed Oct 26 15:06:53 2022
read: IOPS=107k, BW=52.0MiB/s (54.6MB/s)(364MiB/7001msec)
slat (nsec): min=1743, max=20578, avg=1853.32, stdev=84.94
clat (nsec): min=571, max=462377, avg=7237.21, stdev=1091.43
lat (usec): min=6, max=464, avg= 9.09, stdev= 1.10
clat percentiles (nsec):
| 1.00th=[ 6816], 5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7072],
| 30.00th=[ 7072], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7200],
| 70.00th=[ 7200], 80.00th=[ 7264], 90.00th=[ 7328], 95.00th=[ 7456],
| 99.00th=[ 9920], 99.50th=[13632], 99.90th=[15424], 99.95th=[16064],
| 99.99th=[57600]
bw ( KiB/s): min=53220, max=53390, per=100.00%, avg=53313.29, stdev=55.45, samples=14
iops : min=106440, max=106780, avg=106626.71, stdev=111.03, samples=14
lat (nsec) : 750=0.01%
lat (usec) : 4=0.01%, 10=99.02%, 20=0.97%, 50=0.01%, 100=0.01%
lat (usec) : 500=0.01%
cpu : usr=13.17%, sys=34.21%, ctx=746110, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=746110,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=52.0MiB/s (54.6MB/s), 52.0MiB/s-52.0MiB/s (54.6MB/s-54.6MB/s), io=364MiB (382MB), run=7001-7001msec
Steps uses worklets. A description of the worklet used by
the current step is provided below. The description is extracted from the docstring
of the worklet and provided here in verbatim / as-is.
Run fio...
{'description': '\n Run fio...\n',
'extras': {},
'id': '004_plot_fio',
'name': 'plot_fio',
'nr': 4,
'status': {'elapsed': 0.7855701446533203,
'failed': 0,
'passed': 1,
'skipped': 0,
'started': 0.0},
'uses': 'plot',
'with': {'path': 'cijoe-output',
'plot_attributes': 'plot-attributes-512.yaml',
'tool': 'fio'}}